Big sharkdp fan. Ty you for making awesome software that i use DAILY.
bat, fd, hexyl, hyperfine
I'm going to take this moment to remind all of you well-paid engineers that if we each spread $10 a month sponsoring talented software makers like sharkdp the Internet would be a better place.
So many great little tools out there and we should try to support an ecosystem for them.
I owe hours of my life to Andrew Gallant aka BurntSushi's xsv. Nothing else tries to handle splitting long files into N-row chunks [1]. I was using the command line utility, but that's his wrapper around his rust-csv library. So if you need CSV parsing in rust, I strongly recommend this library.
[1] rows included linebreaks so your standard sed/head/tail/something-from-coreutils approach would not work.
I mean there are 131 open issues and some 30+ PRs, so clearly people have some desire for change.
No criticism to the author. He is way more productive than I will ever be, but xsv does appear to be on the back burner. Open source means the author can spend their time how they like and I am entitled to nothing.
readme says
this tool is originally a fork of BurntSushi's xsv, but has been nearly entirely rewritten at that point, to fit SciencesPo's médialab use-cases, rooted in web data collection and analysis geared towards social sciences (you might think CSV is outdated by now, but read our love letter to the format before judging too quickly). xan therefore goes beyond typical data manipulation and expose utilities related to lexicometry, graph theory and even scraping.
Was just going to say, the fd,bat author reminds me of burntsushi (xsv, rg), in terms of the massive benefit they've added to the *nix command-line ecosystem.
Astral is primarily funded by venture capital firms. The head honcho, Charlie, has mentioned a few times that he’d like to develop and sell services that integrate well with Astral’s open-source tools. However, nothing concrete has been established or announced yet.
I would not call that a tutorial on how to write good utility software. It's a quite specific description of how sub(1) was written, and it contains some useful lessons, but not many that would apply to the broad class of "utility software". I think.
in complete agreement, with tools like fd getting more visibility!
we sponsored fd's development a while back and we occasionally sponsor terminal tool authors from time to time at Terminal Trove where we have more tools in the trove. (0)
we're currently sponsoring zellij which I encourage you to check out and sponsor! (1)
It's like so many other projects we're talking about here: it has sane defaults. If you start Zellij for the first time, it shows you menus to do all the things you might want. A big plus is that it supports mouse scrolling by default, and the scrolling works 100% of the time as far as I can tell.
I don't know if it can do everything that tmux or screen can do. I bet it probably can't. But it does all the things I want such a thing to do, and without needing any configuration on my part.
Interesting. When I tried Zellij I found that the menus were too much in my face, compared to tmux/screen more minimal design. I see how this is a good thing when you start using it but I wonder if it gets tiring in the long run, in a clippy sort of way.
But yes it makes sense that this feature makes it superior to tmux for some users. A good case for having diversity in software.
I could see that. I use Emacs with all the toolbars etc turned off. I get it.
I’m not a hardcore tmux user. I just like having persistent sessions on remote machines and bouncing between tabs, things like that. When I want to do something beyond that, with tmux I’d have to RTFM each time. In Zellij, the menu bar lets me discover it easily.
Bottom line: you’re right. I’m glad they both exist!
I'm glad you identified the author -- I'm a big fd and bat fan but didn't know they were created by the same person. I'll have to check out his other tools.
I wish there was an easy way to find people to sponsor whose repos I use (not depend on because every project I use multiplies the same dependencies) but there are tools I use daily that aren't dependencies in my project like wezterm or atuin.
I wish fd and rg would align some of their flags. For example, both fd and rg have a --type, but for fd it means file/directory/symlink etc. For rg it means the file MIME type. Another example is that fd has an --extension flag, and rg doesn't.
Since I believe the correlation of usage of these tools is high, I think they could benefit from having similarly named flags.
To be honest, this is one of the reasons I usually stick with POSIX tools, I’m too old and too lazy to want to learn a whole new set of flags for a whole new set of tools that are very close to but not quite the same as what’s already part of my muscle memory now.
Not taking anything away from the tools that have been written. Just for me, the pain of learning a new tool is greater than the convenience I’d gain from using it.
As the author of ripgrep, I find this unconvincing personally. Have you ever tried to use `sed` with its `-i` flag?
That's because `-i`, while incredibly useful, is not POSIX. So when you say "POSIX tools," what you actually probably mean is, "superset of POSIX tools."
There is some agreement among the same tools as what the options in the superset actually mean, but not always, as is the case with `sed`.
As for whether flags are consistent across different tools, well that's a total crapshoot. `find` uses `-L` for following symlinks while `grep` uses `-R`. Ironically, both `fd` and ripgrep use `-L`, so they are more consistent than the relevant superset-plus-POSIX tooling on at least that front.
To be clear, I mean this somewhat narrowly. Namely
> I’m too old and too lazy
I totally get that. I have a bunch of things I kinda want to learn and do, but just not enough time in the day. But I do still try to make time for these things. I did it for `tmux` (previously, `screen`) and `zsh` (previously, `bash`) and I am very happy I did. Each one cost me about a Sunday and a half, but they've been paying rewards ever since.
> As the author of ripgrep, I find this unconvincing personally. Have you ever tried to use `sed` with its `-i` flag?
I have. But that’s one inconsistency and an edge case I rarely need to worry about.
Plus we are talking about grep here, not sed. I created my own “sed” because I got fed up with dealing different implementations of existing seds. If others use it or don’t use it then that’s their choice, not mine.
> As for whether flags are consistent across different tools, well that's a total crapshoot. `find` uses `-L` for following symlinks while `grep` uses `-R`. Ironically, both `fd` and ripgrep use `-L`, so they are more consistent than the relevant superset-plus-POSIX tooling on at least that front.
UNIX tools are a mess too. My point wasn’t that their flags are more sane than modern tools. It’s that I’ve already committed to memory the flags for the tools I use daily.
> Each one cost me about a Sunday and a half, but they've been paying rewards ever since
I spend my free time writing open source tools that solve problems I run into (like fixing the limitations with existing UNIX shells, and presently writing a new terminal emulator with a ton of features existing ones neglect). So I don’t have time to learn new tools to solve problems I don’t have.
Edit:
I just want add to my previous comment that some of my open source tools do make use of your fantastic Go libraries, such as the Unicode width library.
So I’ve really valued your contributions to open source even though I don’t personally use ripgrep specifically.
I mean that seems pretty reasonable? I find your philosophy a little disappointing, but you do you. But it is reasonable. I think you ignored this part of my comment:
> To be clear, I mean this somewhat narrowly. Namely
>
>> I’m too old and too lazy
>
> I totally get that.
Yeah, I think your explanation is pretty reasonable. I am also old and lazy and still find time to try new tools.
I am not sure I have ever gotten traditional "find" to do what I want on the first try, and I've had a lot of first tries. At some point you have to ask yourself, if I haven't achieved zen in X years with Y tool, maybe the problem is the tool and not me?
I have two young children plus maintain several open source projects (like yourself) and a full time job too.
Time isn’t infinite so I focus my energy on the stuff that makes the biggest impact.
It’s not like I’m ungrateful for your contributions to open source and if you’d look at some of the stuff I’ve been building you’d see we are pretty like minded with regards to modernising the command line.
Oh, no, definitely disappointing! At least to me it is. I totally believe in learning new things for the sake of it, because it sometimes leads to serendipitously good things. Otherwise it's really easy to end up in local optima. Without that philosophy, I would have never built ripgrep.
I'm not trying to convince you to do that. I already told you that your philosophy is reasonable. But I absolutely find it disappointing.
Anyway, yes, I have a kid. And all sorts of other things competing for my time. So I don't get much time to do serendipitous learning. But I try when I can. I cited two examples, but that's over a period of many years. I don't have time to do it often.
> you’d see we are pretty like minded with regards to modernising the command line
I bet we are. Maybe you are a lot more open to learning new things than you are letting on in your comments here. :-)
> I bet we are. Maybe you are a lot more open to learning new things than you are letting on in your comments here.
Oh absolutely I’m open to learning new things.
Just recently I’ve been learning about tmux control mode (eg what iTerm2 uses for tmux integration) because I wanted to bypass tmux terminal emulation for specific escape codes, meaning I can then draw native elements inside a terminal window, such as spreadsheets, images, and code folding, while still having full tmux capabilities.
That was a “fun” ride. I plan on writing a blog about it at some point because there’s some features about tmux that I think others would be surprised to learn.
Oh man that's nowhere near fair! ripgrep is one tiny little command that is mostly compatible with GNU grep. Lots of the flags are the same. And its default usage is really easy: `rg whatever`.
Yeah I started with the shell script route because it felt like the best way to get familiar with it quickly. Then I'd probably go backwards to interactive usage.
The error messages did look very nice.
I have been considering dipping my toes into a less standard shell over the past few years. It is so so so hard to break away from the ubiquitous bullshit that most of us are stuck with. I have no love for the Bourne shell and its derivatives, although zsh is some nice lipstick on a very ugly pig.
The other candidates I'm aware of are fish, nushell and oils. I think nushell is the "most" different, but the last time I tried it, I bounced off of it for reasons I can't remember.
Yeah, switching shell is a major time investment. I couldn’t blame you for sticking with the default, after all, I was initially unwilling to invest time into learning ripgrep :D
And you’re right that it wasn’t a fair a trade asking you to look at Murex in exchange for ripgrep (which is nice by the way!) but I respect that you did take a look nonetheless.
I’m happy you gave ripgrep an option to perform replacements so I don’t have to worry about sed and its lack of a standard way to change files in-place. I realize on my Mac I could install GNU sed, but if I’m going to install an extra utility anyway, why not go with something that is overall nicer?
Hah, well, hate to break it to you, but ripgrep never writes files, only reads them. So its `-r/--replace` option only controls ripgrep's output, but doesn't actually replace data inside a file.
I could swear I used it to do just that a handful of times in the past. But my brain's been a bit scrambled recently so I'm probably misremembering now. I know for sure I made use of the replace option and was happy it was there and that using it felt better than using sed.
I guess it does make sense now that I think about it that ripgrep wouldn't do in-place edits. If ripgrep never performs writes, there's never a chance of a mistake in usage or bug in the software clobbering files in bulk.
sed has to be one of the worst POSIX tools. It sounds simple enough, but everytime I reach for sed it doesn't do what I want, either because it doesn't align with how I do things, or because it just doesn't support doing it (especially multiline replacements for example).
I've switched to sd[1] because it basically just works as I expect every time.
ripgrep solves a really annoying part of the unix toolbox: inconsistency in how to correctly search a bunch of files for a string. Are you supposed to `find -name -exec`? Or are you supposed to `find -name | xargs -n 1`? Oh, but files can have spaces, so you might try `find -name -print0 | xargs`, but careful—`-print0` is not POSIX and you won't find it on some unixen! (let's not even discuss locate vs slocate vs mlocate.... ugh! Files are worse than everything but all the other options.)
this is what tab completion and tldr is for. 99% of use cases are well covered and clear by the name of the flag, and a good CLI tool will make that easy to understand. A quick example and self-explanatory flags with tab-completion is all you need. Then if you ever have a more complicated use case, you can grep through the man page.
its legit as simple as "fd -e png -x optimize-png {}" the only thing I dont like about fd is that for some reason it kind of forces you to do 'fd . Downloads' if you just want everything in "Downloads" which equates to 'fd {pattern} Dir1 dir2" I wish you could omit the pattern sometimes.
I’ve been thinking of adding AI completion into my CLI tools but not really sure how to implement it in a non-intrusive way. So I’ve got a few questions, if you don’t mind sharing:
What’s the workflow like for AI shell completion?
How does it know which flag to complete for you? Do you write a description in native language (eg English) and it completes the entire command line? Or is it more one flag at a time?
Yeah this annoys me even though I'm a daily user of both fd and rg. What makes it more confusing is that many of the flags DO align - or partially align.
For example, I'm used to glob patterns but the glob flag (-g) works differently in fd and rg. I think that fd's -g flag does not use "full-path" globbing while rg's -g does (or the other way around). To get fd to use rg style globs, it also needs the -p flag, which rg also recognizes but it has a completely different meaning for rg and has nothing to do with how globbing/filename matching works.
I guess I'm used to the warts at this stage, like I had gotten used to the warts on find and grep all those years ago.
Difficult or impossible to fix these inconsistencies at this stage without breaking backward compatibility.
It's a tricky balance. To use your --type example, it isn't consistent with rg, but it is mostly consistent with find. And fd'a --type option is much more useful for fd than an equivalent would be for rg. It doesn't make a lot of sense to filter the files to grep to directories, or sockets, but that is useful if you are searching for file names, or even just all files of a certain type. Conversely, rg's --type option isn't quite as useful for fd, because fd is already matching a pattern against the file name, so you can easily just add the appropriate extension to your search pattern. Or use the --exyension flag.
One reason I haven’t picked up any of these newfangled Rust tools like bat, exa, or fd is that I can barely remember the options for the originals.
For me, anything that isn’t a drop-in replacement for the OG tools isn’t worth the friction. I use ripgrep inside VS Code but vanilla grep on the command line because of years of muscle memory.
That said, I don’t care what language a tool is written in as long as it works. One of my favorite Unix tools is GNU Stow, and it’s written in Perl. Even if these Rust tools were drop-in replacements, I probably wouldn’t bother installing them manually. As a user, the speed improvements and memory safety don’t really matter to me.
There are other languages, like Go, where memory safety is guaranteed as well, and Go’s performance is more than adequate for tooling—with the added benefit of getting more engagement from the community. So I’m not entirely convinced by this “Rust is the savior” narrative.
That said, if macOS or Ubuntu decided to hot-swap the OG tools with Rust alternatives that behave exactly like their predecessors, I probably wouldn’t complain—as long as it doesn’t disrupt my workflow.
>One reason I haven’t picked up any of these newfangled Rust tools like bat, exa, or fd is that I can barely remember the options for the originals.
But that's exactly the reason to use the newer tools -- they just make more sense -- especially fd over find. I've been using UNIX for over thirty years and find just never clicked with me.
fd is probably better for most tasks, but sometimes it seems more cumbersome than find. E.g., to delete all files inside a cache directory, this is the simplest syntax I could find:
fd -t f -X rm {} \; ^ cache
Which makes me really nervous, so usually I fall back to using find:
find cache -type f -delete
Maybe this is foolproof for me only because I’ve been using find for decades. Is there a version of this for fd that inspires more confidence?
Which reads as find "any file", "matching .", "in directory cache", then "execute rm -- followed by an argument list of all found files".
This ensures even if you have filenames starting with - they won't be interpreted as options for rm. For even more sanity of mind you may want to turn on -a for absolute paths, although I don't see an example right now where using relative paths would go wrong.
It absolutely does not matter what language this tool is written in. That goes for any tool. If it’s better, use it.
In this case, fd is far superior to “find” in almost every way. Sane defaults, wayyy faster, easy options (just use cht.sh if you can’t remember) To me, there is no reason to ever use “find”. If I’m on a new system, I just install fd and carry on.
> It absolutely does not matter what language this tool is written in. That goes for any tool.
Eh, there are a lot of tools where it actually does kind of matter. I suspect for a lot of invocations of tools like `fd` and `rg`, they'll be done before an equivalent written in java has even had its JVM spin fully up.
There's _tons_ of Java software, but it somehow never managed to make a dent in the CLI space.
> To me, there is no reason to ever use “find”. If I’m on a new system, I just install fd and carry on.
I guess I should finally have a look at how to replace my `find $path -name "*.$ext" -exec nvim {} +` habit … turns out it's `fd -e $ext -X "nvim" "" $path`
Another reason to at least learn the default tooling is that often I find myself SSHing to another machine which has only the barest of default packages installed (often busybox, sometimes just a stripped-down docker container).
If I didn't know how to use "find" and "grep" (though I prefer rg) then I'd be at a disadvantage in these situations. Also command-line git.
It's why I learned to use Vim well, though I daily Emacs.
>> For me, anything that isn’t a drop-in replacement for the OG tools isn’t worth the friction.
"The uutils project reimplements ubiquitous command line utilities in Rust. Our goal is to modernize the utils, while retaining full compatibility with the existing utilities. We are planning to replace all essential Linux tools."
If you're already proficient with grep, find, etc - there's not much reason to switch. As I said elsewhere:
I never managed to use find because I always had to look up command line arguments. I would always find a different way to solve my problem (e.g. Midnight Commander).
There are a few reasons you might still want to switch. In fd'a case:
- It respects .gitignore files (as well as similar .fdignore files that aren't git specific), which can help you find what you care about without a lot of noise, or having to pass a lot of exclude rules to find
- it can search in parallel, which can significantly reduce latency when searching large directories.
However, there are also reasons you might want to keep using find:
- fd can't do everything find can. fd is intended to replace the most common use cases with a simpler interface, but there are many less common cases that require finds greater flexibility. In fact, I still use find sometimes because of this.
What about https://zolk3ri.name/cgit/zpkg/? A lot of improvements have been done behind the scenes apparently (rollback, proper states, atomicity, etc.), but I am not sure when he is willing to publish.
I personally use it as-is when I am compiling stuff myself and the ZPG_DST is ~/.local/. It works well for keeping track of programs that I compile and build myself.
It resonates with me wrt muscle memory and ubiquity of “standard tools” that come pre-installed in majority of *nix distros including macos.
But there is a big BUT! Lately I have to use grep/find huge nested dirs and found rg to be an order of magnitude faster. Had to get myself comfortable with retraining the muscle memory. Worth the effort.
Some of these new shiny tools are meh for my taste. Delta for instance. Or helix the editor. But it is personal. Overall I love the competition. It seems like industry once full of innovators and tinkerers is lacking some shake up.
I understand the point about muscle memory but I think that was more of a concern in the days before we had it easy with instant internet answers and now LLMs (eg GitHub copilot command line) doing our boring thinking for us.
Is anyone else bothered by the fact that by default it ignores a lot of folders? I use `find` when I'm like 'I just want to know where on my system this is, wherever it might be'
I know fd has options to not ignore things, but I can never remember them, so I just go back to find because I know it'll search everything.
I actually prefer it. It's very similar to ripgrep's default search. I do occasionally want to find something in a .gitignore, or other hidden directory, but I don't mind taking the time to `rg/fd --help` and add the flag to include hidden directories.
I completely dumped the windows search and only use voidtool's Everything when I am on a Windows box.
It can search multiple indexed NTFS drives in miliseconds. Indexing is usually a few seconds since it works directly on the NTFS structures.
(and it integrates with Total Commander)
Same, this is why I haven't fully converted to lots of these newer tools. If I have to remember a bunch of special flags to make them look at the files I need them to, their advantage is lost. I'm better off using the originals and solidifying those flags in my muscle memory since at least those tools will be on any system I use. I do use ripgrep on occasion but not very often.
That’s a feature, and one of the reasons I prefer it. When I want to find a file in a git repo, say, there’s no need looking inside .git most of the time. Sometimes there is: if I don’t remember `fd -u`, there’s good old find there for me. But that’s almost never what I want to do, so fd’s defaults are sensible for me.
I’ll have to try this out. I admit that most of my uses of find looke like
find . | grep what_i_am_looking_for
Because I can never remember how finds arguments work. I like the integrated xargs like behavior as well.
One thing I did not see in there was how fd handles symlink directory traversal? Searched the whole readme for it, and only found options to match on symlinks or not.
Hm, I feel like I have roughly the exact same functionality with zsh and some various plugins, including the meta editing experience shown in your demo video ($EDITOR) via Kitty and/or WezTerm.
With some effort it’s possible to bend zsh to any workflow, but there’s a lot to be said for having sane defaults.
The other value add for alternative shells like my own is better support in the language for handling structured data (is not treat everything as a dumb byte stream).
Ultimately though, productivity is all about familiarity and if you’re also super efficient in zsh then I’m not going to try and convince you that some shiny new tool is better.
I’m finding that I use locate more and more for this sort of thing. Of course, it produces a lot more output, because it looks at the whole system. But, it is still much faster than find because it runs on some kind of index or something, and it is easy enough to grep out the directory I’m interested in, if needed.
locate is nice, but I think that on most distros its index is only updated once/day (unless you adjust the cron job that updates it more often). Most of the times I'm trying to find something, I haven't modified it recently, but it can definitely lead you astray.
One can just run 'sudo updatedb' to refresh it. Super fast if it's already on a schedule. I think mine is set to update every system update, so I just have a habit of running that before I use locate each time.
In that case, you might as well just use grep -r, or its alias rgrep. And then remember that it supports --include=GLOB, --exclude=GLOB, and --exclude-dir=GLOB.
Not the same thing, but thank you. I've been using "find . -type f -exec grep -i {} /dev/null \;" without looking for a better solution and here it is.
I use the heck out of fd daily and it’s in my standard installation on new machines. I’ve used find for years and it’s great in its own ways, but convenient ergonomics isn’t among them.
I’m 100% on board with this recent-ish trend toward new replacement utilities that may or may not retain all of the flexibility of the originals but are vastly easier to use for common cases.
As much as i love new rust cli tools, `fd` ended up in the same bag as tar and ln.. I can never retain how to use it. I don't blame anybody .. I'm just witness that I always have to re-read the man from scratch to get anything done. And I'm not a gnu find lover.. with all its idiosyncracies.. somehow it registers better.
Why would you love them just because they're in rust?
I'd like to praise the author of this fd for not having "Rust" all over the web page or in the HN link, actually.
The Rust inquisition would get far less pushback if instead of threatening people that god will kill a kitten every time you use a non-kosher programming language, they'd promote software that improves (even opinionated improves, like this fd) on existing tools instead of having its main feature "but it's written in Rust!".
That is literally what people are doing. “Written in Rust” has become a pretty convenient shorthand for “this was written by someone who cares deeply about quality, and you can guess that it will be extremely fast, relatively free of bugs, and use as many cores as you want to throw at it”.
While you’re out here complaining, the Rust community has built a truly impressive array of improved, human-focused command line tools (rg, bat, fd, hyperfine, delta, eza, and on and on) and a bunch of best-in-class libraries for building tools (regex comes to mind).
While mostly true, there's a large problem with lots of these Rust CLI tools (and I use some of them, don't misunderstand me): they often discard the UNIX philosophy in favour of newbie friendliness and glitter.
Feels like if someone did a RIIR on ImageMagick, it'd be able to talk with Instagram and Flickr by itself or process RAW files like Darktable. Would probably have some kind of tagging system too.
I used a similar wrapper for years before I wrote ripgrep. But now all of those wrappers are gone because ripgrep covers all of their use cases (and then some). I'm not a newb and I don't really care about glitter. So I don't know where you're getting your condescension from personally.
GNU grep discards the Unix philosophy in all sorts of places too. Like, why does it even bother with a -r flag in the first place? Did people back then not know how to use xargs either? I mean, POSIX knew not to include it[1], so what gives?
What's actually happening here is that what matters is the user experience, and the Unix philosophy is a means, not an end, to improve the user experience. It's a heuristic. A rule of thumb to enable composition. But it doesn't replace good judgment. Sometimes you want a little more Unix and sometimes you want a little less.
Besides, you can drop ripgrep into a shell pipeline just like you would grep. All that Unix-y goodness is still there.
> GNU grep discards the Unix philosophy in all sorts of places too.
Who ever claimed that GNU was a proponent of the UNIX philosophy? Certainly not me.
> What's actually happening here is that what matters is the user experience, and the Unix philosophy is a means, not an end, to improve the user experience
Yes and no. The expression "user experience" implies that all users are the same and benefit from the same philosophy of software interface design.
> Besides, you can drop ripgrep into a shell pipeline just like you would grep. All that Unix-y goodness is still there.
The "Unix-y goodness" isn't only defined by what it can do, but also what it can't do. I don't want to turn this into a suckless rant, so I won't. Also, not having a mode giving it POSIX grep compatibility really flies in the face of that claim: I want my scripts to work on BSD, MacOS and even Solaris/AIX without having to install stuff.
------
I know I may sound very smug but believe me when I say that I really like some of these utils and the care put into them (and I know you're the author of ripgrep) and that I'm no UNIX OpenBSD user (Plan 9 exists for a reason, and I favor CL over C/Go any time of the day). But I really see them as optimizing for/targeting the VSCode/Zed crowd, instead of the vim/emacs one, if you know what I mean.
> Who ever claimed that GNU was a proponent of the UNIX philosophy? Certainly not me.
Nothing was specific to GNU in that comment. `-r` is available in BSD grep too. You are being needlessly pedantic and avoiding the actual points being made while doing so.
> Yes and no. The expression "user experience" implies that all users are the same and benefit from the same philosophy of software interface design.
No, it doesn’t. You have literally made that up out of whole cloth, but it isn’t how anyone else I’m aware of defines “user experience”.
As BurntSushi pointed out, ripgrep is even designed to adapt to different means of use: as a standalone command, as a utility in a pipe, and others. Improving the usability for one use-case isn’t necessarily at the cost of others.
> I want my scripts to work on BSD, MacOS and even Solaris/AIX without having to install stuff.
So your very reasonable desire to stick with POSIX for shell scripts… means we shouldn’t ever make new tools? I’m genuinely failing to understand your point here.
In fact, having a “grep compatibility flag” would very literally be the opposite of the UNIX philosophy you’re holding up elsewhere. ripgrep isn’t grep. It isn’t trying to be grep. It’s something new and different and (in some use-cases) better. The old thing still exists and if you want to use it for whatever reason… just do that?
> Nothing was specific to GNU in that comment. `-r` is available in BSD grep too. You are being needlessly pedantic and avoiding the actual points being made while doing so.
Where's the actual point in "others do it too"?
> No, it doesn’t. You have literally made that up out of whole cloth, but it isn’t how anyone else I’m aware of defines “user experience”.
That's not a question of definition, but of use in an English sentence. If you say "this is good for user experience", you implicitly group all users together. I say some users prefer tools that do only one thing. ripgrep itself acknowledges that "despite initially not wanting to add every feature under the sun to ripgrep, over time, ripgrep has grown support for most features found in other file searching tools". And that's fine! But not for everybody.
> As BurntSushi pointed out, ripgrep is even designed to adapt to different means of use: as a standalone command, as a utility in a pipe, and others. Improving the usability for one use-case isn’t necessarily at the cost of others.
If I like the "finding files with auto filtering" part of ripgrep, how can I use it to feed another grep or even another tool? If it had been designed with the aforementioned philosophy, it would have been "ripfind" + "ripgrep" and maybe a "ripfg" pretty interface on top.
> So your very reasonable desire to stick with POSIX for shell scripts… means we shouldn’t ever make new tools? I’m genuinely failing to understand your point here.
> In fact, having a “grep compatibility flag” would very literally be the opposite of the UNIX philosophy you’re holding up
See my other reply: I love new tools, but "UNIX-like goodness" is also about portability (as in SUS), not just the philosophy. More of a misunderstanding than anything else, really.
A flag would be impossible, but introspection based on argv[0] (like Busybox) wouldn't be that far-fetched. But yeah, clearly ripgrep isn't trying to replace only grep, that was a mistake on my part.
As an addendum, the "UNIX philosophy" had a lot of crap parts (e.g. bytes as universal interchange format) and UNIX itself is a very poor implementation of said philosophy (after all, g/re/p itself is pointless porcelain over (s)ed, heh), I'm not sure discussing it seriously without specifying which part (like "do one thing") is very productive.
Happy this was thought out! Still everything in one tool, but glad modularity was truly considered.
> Except for Windows.
Of course POSIX portability doesn't apply to OSes which intentionally ignore it. I think I have another flame appreciating guy on the other side of the conversation =)
It just means that your claims about portability are, in the real world, far less useful than is imagined. I ship software to Windows users and the "portability" of POSIX doesn't help there. Doesn't matter why to me. It just means that I can't depend on POSIX for the "portability" benefits that its proponents seem to cite so enthusiastically.
> But I really see them as optimizing for/targeting the VSCode/Zed crowd, instead of the vim/emacs one, if you know what I mean.
At this point it sounds like you're making a purely aesthetic/cultural judgment, rather than a technical one. ripgrep works just fine with emacs; I have "C-c p s r" bound to "run ripgrep in current project" and use it almost every day.
OK, so you phrased your criticism very narrowly toward "lots of these Rust CLI tools," but in actuality, your criticism applies way more broadly than that. Likely to just about all of GNU's tooling, for example. And probably BSD's and busybox's tooling too. Even busybox isn't above implementing POSIX+some, and AFAIK none of them provide a strict "POSIX only" mode.
Which is fine, but now your position is more "old man yelling at clouds" than it is something specific about the Rust tools. (And to be clear, I'm not above yelling at clouds now and then.)
> Also, not having a mode giving it POSIX grep compatibility really flies in the face of that claim
No... it doesn't? ripgrep being compatible with the Unix philosophy is not the same as it being POSIX compatible. I didn't say you could use ripgrep in any place you could use grep in a drop-in compatible way. I said you could use it in shell pipelines. As in, it inter-operates with other tooling in a Unix-y way.
And besides, I don't know of any grep that has such a mode. How do you know you aren't using a grep flag that isn't available in other grep implementations?
> But I really see them as optimizing for/targeting the VSCode/Zed crowd, instead of the vim/emacs one, if you know what I mean.
Nope, I don't. I've been a vim (now neovim) user for decades. And neovim even uses ripgrep by default over grep if it's installed.
> OK, so you phrased your criticism very narrowly toward "lots of these Rust CLI tools," but in actuality, your criticism applies way more broadly than that.
Obviously. The topic was Rust CLI tools, that's why my post focused on them. Indeed, my criticism is broader.
> Likely to just about all of GNU's tooling, for example. And probably BSD's and busybox's tooling too. Even busybox isn't above implementing POSIX+some, and AFAIK none of them provide a strict "POSIX only" mode.
This isn't about POSIX or not POSIX, this is about introducing yet another pure convenience feature for something that can easily be achieved with other accompanying (guaranteed to be on the concerned platform) tools. POSIX doesn't create features anyway, it's about describing things that have already become standard; the Austin Group bug tracker is quite explicit about this.
> No... it doesn't? ripgrep being compatible with the Unix philosophy is not the same as it being POSIX compatible.
Well, just a question of interpretation of "Unix-like goodness". UNIX is both a philosophy and a portability standard (since POSIX == SUSv4). And as I said, doing one thing and doing it well is another facet of that philosophy; finding files != searching in them.
> And besides, I don't know of any grep that has such a mode. How do you know you aren't using a grep flag that isn't available in other grep implementations?
Compatibility != strict rejection of anything else, quite obviously.
> Nope, I don't. I've been a vim (now neovim) user for decades. And neovim even uses ripgrep by default over grep if it's installed.
Which is why I said vim instead of neovim (even if neovim is clearly better).
------
I think you misunderstand my position because I like a bit of snark: compromising on the "do one thing" part of the UNIX philosophy isn't necessarily something horrible, but it's not an imaginary issue either. I could give other examples (like how the pretty clean xsv became https://github.com/dathere/qsv) but it really doesn't matter because I think it's more of a popular CLI "apps" thing than Rust alone, even if there's community overlap.
I can no longer see any coherent point you're trying to make.
My point is that you are treating an ideal ("do one thing and doing it well") as an end instead of a means. Moreover, you are treating it as if there is any actual choice in the matter. In reality, the tooling most of us use on the command line violates the Unix philosophy in some way or another. So the question is not "should I adhere to the Unix philosophy?" but "what makes a good user experience?"
I don't see why my point is that hard to see: the "do one thing" part of the UNIX philosophy is a good ideal to aim for, even if often ignored in favour of a more convenient kitchen sink approach. I then just mentioned that a good majority of these useful Rust tools (I really like hyperfine and oxipng, for example) don't seem to care much about it.
I’ve also been guilty of insisting on this, even when, as you point out, most tooling doesn’t strictly follow it anyway. I think it’s an overreaction from being irritated at seeing app after app fall to “now including AI,” or just pivoting for new features because it wasn’t making money fast enough.
FWIW, I’ve been using *nix for about 25 years, and had zero problems adapting to ripgrep. You kept -i and -v, which honestly covers 90% of common usage.
Sorry for the late reply. I just wanted to say that I don't think AI has anything to do with this. I've been seeing this sort of complaining, goal post shifting and confusing means with ends for probably decades now. I mean, AI was around then too, but not like it is today, which is what I assume you meant. Even when I release ripgrep ~9 years ago, AI wasn't what it was today, and I saw the same sort of complaints.
I don't usually like doing this, but if I had to step out of my wheelhouse and guesstimate, I think it's probably just some form of nostalgia reasoning. People remember the "simpler" times and just want to have it again. But of course, the "simpler" times are less a reflection of reality and more a reflection of your perception of the world. IMO anyway. (I have the same sort of inclination, but I try to recognize it for what it is: personal bias. Others seem unable to do this and mistake this personal bias for objective reality.)
A tool doesn't need to support locales to be used in a shell pipeline. The Unix philosophy and shell pipelines were born long before POSIX locales. I even specifically contrasted this with POSIX compatibility. So it seems to me like you're disingenuously interpreting my words at this point.
That isn't true. ripgrep will happily pass through data it gets as-is. It just itself doesn't support locales. Commands do not need to have locale support to be used in shell pipelines. `rg foo | rg -v bar` is an example that has nothing to do with locales but demonstrates that ripgrep can be used in shell pipelines.
If foo prints $'O\u011Fuz' and LC_ALL is set to a valid UTF-8 locale, bar will receive what foo printed. If foo prints $'O\xF0uz' and LC_ALL is assigned a valid ISO-8859 locale, again, bar will receive what foo printed; because foo, bar, and grep are compatible. You can't have that with ripgrep; you'll have to modify the shell script in between or worse, manually parse LC_ALL and hope ripgrep supports the encoding
First of all, that doesn't invalidate literally anything I said. I have never claimed that ripgrep supported locales. I only said that it could be used in shell pipelines. That doesn't mean it can be used in all shell pipelines in exactly the same way that grep can. I even clarified this explicitly. Which is why you are a troll.
Second of all, you're just wrong. ripgrep uses UTF-8 by default, but you don't have to use it. And when you switch your locale to ISO-8859 (who uses that!?!? why does it mater!?!?), you are no longer emitting UTF-8. As you obviously know or else you wouldn't have been able to come up with the example.
As has been pointed out elsewhere in this thread, you can disable Unicode mode in ripgrep:
I'm sure you can get creative. :-) You can set an environment variable to control the encoding, expose a flag or any one of a number of other things to control the encoding. Either way, now you're just shifting the goalposts.
You've also continued to ignore my most substantive rebuttal: that a specific example where ripgrep is not compatible with grep or doesn't behave the same doesn't mean it can't be used in shell pipelines. I use ripgrep in shell pipelines all of the time. As do many others. Literally nothing you've said has invalidated anything I've said. All you're doing is finding things that some implementations of grep can do that ripgrep (intentionally) cannot do in exactly the same way. But that's fine, because ripgrep was never, isn't and will never be compatible with grep: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...
So if you need grep compatibility get a fucking clue and just use grep.
> You can set an environment variable to control the encoding
Yes, that's what LC_ALL is. Every other tool understands it except ripgrep. Even if you parse it by hand, there's no guarantee that ripgrep will support the encoding.
Apples vs. oranges. Busybox is a suite of minimal tools for low-resource systems, lack of QoL features you wouldn't need in such an environment is its selling point. Besides you won't see anyone claiming individual busybox tools can be used in shell scripts just fine.
It's easier to admit ripgrep is rather a supplementary tool that is incompatible with the tooling your typical Unix-like operating system provides out of the box than a "modern replacement" for grep and move on.
So you lied. Now you've shifted the goalposts after being called out on it. And ripgrep has no problems being used in shell pipelines. What ripgrep has a problem doing is being drop-in compatible for grep, because that isn't its goal.
> "modern replacement" for grep
I have literally never called ripgrep this. So more lies and straw-manning from you. ripgrep's repo neither positions itself as a replacement or as "modern." And in the FAQ about "replacing" grep, it makes a nuanced claim that accounts for all of this. It specifically says you should use grep when you need the features of grep that aren't in ripgrep.
It's not strange because ripgrep doesn't understand non-UTF-8 data (unless there's a UTF-16 BOM, in which case, ripgrep will automatically understand it). But you can tell it to:
The person you're responding to has been trolling in this thread (and others) by twisting words and claiming multiple false things. When I've fixed their errors, they don't acknowledge them as mistakes and just keep on twisting words.
I personally appreciate the newbie friendliness and glitter, and even the "make the common case easier at the expense of composability" tradoff, though I think tlmost of these tools are still relatively composable.
The ecosystem enables focusing on the task at hand. It's great that you don't need to get side-tracked at every single step. There's many high quality libraries.
> Why would you love them just because they're in rust?
That's not it. People love the new rust tools because they're great tools with sane defaults. They happen to be written in Rust most of the time and that's it so people use this to describe them.
That's a bit extreme. I'm not who you replied to, but IME, things written in rust tend to be fast and sane. All else equal, "written in rust" is a positive signal to me.
Surprisingly, Go is great for CLI tooling. It may not have the insane speed that carefully planned and written Rust does, but it's very easy to write and be performant without even needing to go to great lengths to optimize it.
I generally say that anything under 500ms is good for commands that aren't crunching data, and even Python CLI tools can come in under that number without too much effort.
Unfortunately, Python CLI startup time is correlated with how many files the interpreter has to interpret, which usually takes effort to combat, so larger CLIs that have a lot of functionality and thus files always seem to slow down.
The Azure CLI is a good example of this. It's been a year or two since I used it, so maybe it's improved since then, but it used to take a second or two just to run `az --help`, which infuriated me.
If you own a slow Python CLI, look into lazily importing libraries instead of unconditionally importing at the top of each Python file. I've seen that help a lot for slow Python CLIs at $DAYJOB
Of course, then we get the question, ok, why not use a program that just provides those options? But, I think all the extra functionality of tar is nice to have sitting off there in the background; I’d have to look it up if I actually wanted to use it, but it is all there in the documentation, and nicely compatible with the files produced by my memorized commands.
The one that always trips me up is that `ln` is “dumb” about how it handles the first argument.
If the first argument isn’t an absolute path, it must be relative to the second argument, and not to pwd.
ln ./foo ./bar/baz
./bar/baz will be a symlink whose literal contents are `./foo` so it will look for foo in the same directory (bar), rather than in bar’s parent directory.
This is totally backwards from how other utilities behave. Which is completely understandable if you know what it’s doing under the hood. But it is counterintuitive and surprising.
Absolute paths will break if, for example, the target and source are in the same directory but you move the directory.
Sometimes you semantically want absolute paths (I want to symlink a thing to /bin/foo), sometimes you want relative paths (I want to symlink a thing to something in a nested directory).
This is what made the “ln” argument order click for me as well.
Another piece of this parallel is that (with cp and mv) you can omit naming the destination file - you often just name the directory:
cp /other/dir/foo.txt .
The corresponding shortcut with ln is not naming the destination at all, and the symlink is created in the working directory:
ln -s /other/dir/foo.txt
Both of the above are my most common usage patterns, and the abbreviation of the second argument in both helps reinforce the parallel between cp, mv and ln.
curl https://source.code/src.tar.gz | gunzip | tar -x
That's all.
The core operations taking up only this much headspace leaves the door open to slowly start to retain all its other useful options. For example, -v, which means verbose almost everywhere, thankfully means verbose here too, so I add that when I want verbosity.
Similarly, I find it easy to retain -l which lists the contents. I do this quite often for stuff from the internet -- I don't like it when I untargz in ~/Downloads and it untargz's it in the same directory instead of doing everything inside a parent directory (preferably with the same name as the archive)
Bonus: separating out the gzip like this makes it easy to remember how to use zstd if you want to, just pipe to zstd instead! "Unix philosophy" and all that.
I agree with you about `ln`, I can never seem to remember if it's source or target that comes first.
Good to use concurrency, predictable impact on all of the various timings. Different results vs. find for the same term (assumedly) are unexpected? Did I mess up the translations?
One of those essential tools, that's not ever included with GNU/Linux by default, but I always install almost immediately along with ripgrep, bat, fzf, htop.
It supports all the common grep options and you can split view in the -Q mode and it is smart enough to open your $editor at the line number it's talking about.
Try it, this is the workflow I've been waiting for.
Agreed! Curious what your base OS is. Mine has been Ubuntu LTS for years, but now I find myself installing so many custom tools like these I'm thinking I might want to move to NixOS or Gentoo or something like that instead. I just don't want to lose out on stability and great hardware support.
I would recommend trying home-manager, or just plain nix profiles before going all-in on NixOS, it's very easy to add Nix on top of any Linux (and MacOS to an extent).
This way you still have your tried and true base system and you can manage things somewhat like you're used to (and use venv and whatever bullshit package manager everyone invents), a lot of common tools will work poorly on NixOS (venv, npm...)and while they have Nix alternatives that are "better" it's DIFFERENT.
I run NixOS on desktop and laptop but I wouldn't recommend starting with it, you can benefit from all packages on any distro by just installing Nix.
Also home-manager adoption can be incremental.
ADHD end note: home-manager is perfect for CLI tools installation and config. You must unconfigure "secure path" in sudo though otherwise your CLI tools don't work with sudo
My favorite way is to just have this in my zsh aliases.
# Find file by prefix (ignore case).
# Usage: 'f' or 'f myfile' or 'f myfile.txt \etc'
# or f '*css'
function f() {
find ${2:-.} -iname "${1}*" 2>/dev/null | grep '^\|[^/]*$'
}
# We use noglob, so zsh doesn't expand characters like "*"
# and so we can do e.g. f *css
alias f='noglob f'
I just released an Alfred workflow[0] for searching a group of user-defined directories, that heavily relies on fd under the hood. Thank you @sharkdp for this amazing tool, I use it every day and it is wonderful.
Hyperfine is another underrated gem that is just so great.
There are many completely unnecessary "reworks" of classic UNIX stdutils that add very little to no value, and instead pitch themselves purely on "written in {lang-the-dev-likes}" or being "blazing fast" or something similar.
`fd` isn't one of those.
`fd` is amazing.
It is simpler, faster and easier to grok than `find`, the commands syntax is a straight up improvement, it comes with sane defaults which makes the easy things and common usecases simple, while the harder ones are way more accessible. It uses colored output in an actually useful way, its manpage is more structured and easier to read...
and MOST IMPORTANTLY
...it is capable of running commands on search results with parallel execution out of the box, so no need to get GNU parallel into the command.
THIS is the kind of renewal of classic command line utilitis we need!
Literally the only pain point for me, is that it doesn't use glob-patterns by default, and that is easily fixed by a handy
Theoretically yes, practically not really (any more). NVMe cards are fast enough that a single core traversing the file system can actually be a chokepoint, so parallelisation helps a lot here.
I also should have made it clear that my comment also wasn't so much about the search (although the parallel search is absolutely a nice-to-have)...it was about the `-x, --exec` being automatically in parallel.
A common usecase is to find all files of X criteria, and then perform the same operation on all of them, e.g. find all session logs older than N days and then compress them, or convert all wav files in a directory tree to mp3
If the operation is computationally expensive, using more than one core speeds things up considerably. With `find`, the way to do that was by piping the output to GNU parallel.
With `fd` I can just use `-x, --exec` and it automatically spins up threads to handle the operations, unless instructed not to.
Depends greatly on the use case. If you're finding all the gifs in a directory and converting them to pngs, it may be the case that CPU is hammered harder than IO. There are surely counterexamples, too.
A lot of developer machines are running expensive NVME SSDs, which perform well enough the disk isn't so much of a choke point. If you run on spinning rust, YMMV
I find these tools super awesome, but i never use them beyond trying them out once, because they don't usually come with coreutils or something like that.
Haven't found a way to use these tools in a user-local way.
For my own homemade tools, i put the binary in my dotfiles and they end up in .local/bin and i can use them by default. I can do that because I don't need to update them.
I use fd all the time. I admit it, I couldn't get the muscle memory right for find. One day I'll be on a bare Linux installation where I can't cargo install whatever I want and I'll have to ask ChatGPT to generate my find commands.
I like `fd -o {user}` and `fd --newer {time}` to find recently changed files and owned by others.
I like it but the defaults are super annoying. I'm always trying to find things I can't otherwise but it defaults to ignoring git dirs and dot files, literally the thing I search for the most
After using nushell for a few month it's really hard for me to go back to the old ways. A million of different tools all with different API and outputs are just not the way to go
I guess you're right. Isn't the former more useful in general? I find myself reaching for `:Rg` in Vim much more often than `:Files` or `:GFiles` courtesy of fzf.
The other day I was searching for a PDF I downloaded three years ago from a certain company.
Simply ran `find ~/Downloads -iname '<company-name>'` and it immediately popped up.
rg would not have helped me there. Even if I used something like rga, it would have taken significantly longer to parse the contents of every file in my home directory.
Ok, but like, in practice this is a pretty weird edge case. It's impractical and usually worthless to have filenames that can't be described using the characters on a keyboard.
Disagree, filesystems provide for both services and people... this is an imposition. I, a mere human, may need my tools to wrangle output generated by other software that has never once used a keyboard. Or a sensible character/set, bytes are bytes
File extensions - or their names - mean absolutely nothing with ELF. Maybe $APPLICATION decides to use the filename to store non-ASCII/Unicode parity data... because it was developed in a closet with infinite funding. Who knows. Who's to say.
Contrived, yes, but practical. Imposing isn't. The filesystem may contain more than this can handle.
My point is that it's such a weird edge case in the first place that the chances of you needing to use a tool like fd/find in this way is vanishingly small. I agree with the general issue of treating file paths as encoded strings when they are not. Go is the worst offender here because it does it at the language level which is just egregious.
Regardless, the point is moot because `fd` handles the filenames gracefully you just need to use a different flag [0].
It's also what made Python 3 very impractical when it orginally came around. It wasn't fixed until several versions in despite being a common complaint among actual users.
Any of them. File names are in the vast majority of cases human readable in some character encoding, even UTF-8. You would be hard pressed to find a keyboard/character code that has characters that aren't represented in Unicode, but it doesn't matter, just honor the system locale.
Common, but rather stupid. Filenames aren't even text. `fd` is written in Rust, and it uses std::path for paths, the regex pattern defaults to matching text. That said, it is possible by turning off the Unicode flag. `(?-u:\x??)` where `??` is a raw byte in hex. E.g. `(?-u:\xFF)` for OP. See "Opt out of Unicode support[1] in the regex docs.
IMHO, the kernel should have filesystem mount options to just reject path names that are non-UTF-8, and distros should default to those when creating new filesystems on new systems.
For >99.99% of usecases, file paths are textual data, and people do expect to view them as text. And it's high time that kernels should start enforcing that they act as text, because it constitutes a security vulnerability for a good deal of software while providing exceedingly low value.
So just turn off support for external media, which could possibly be created on other platforms, and all old file systems? Legacy platforms, like modern Windows which still uses UCS-2 (or some half broken variant thereof)?
While I support the UTF-8 everywhere movement with every fiber of my body, that still sounds like a hard sell for all vintage computer enthusiasts, embedded developers, and anyone else, really.
As I said in another comment, you can handle the legacy systems by giving a mount option that transcodes filenames using Latin-1. (Choosing Latin-1 because it's a trivial mapping that doesn't require lookup tables). UCS-2 is easily handled by WTF-8 (i.e., don't treat an encoded unpaired surrogate as an error).
The reality is that non-UTF-8 filenames already break most modern software, and it's probably more useful for the few people who need to care about it to figure out how to make their workflows work in a UTF-8-only filename world rather than demanding that everybody else has to fix their software to handle a case where there kind of isn't a fix in the first place.
Oh, I agree that "text" isn't well-defined. The best I can come up with is that "text" is a valid sequence of bytes when interpreted in some text encoding. I think that something designed to search filenames should clearly document how to search for all valid filenames in its help or manual, not require looking up the docs of a dependency. Filenames are paths, which are weird on every platform. 99% of the time you can search paths using some sort of text encoding, but IMO it should be pointed out in the man page that non-unicode filenames can actually be searched for. `fd`'s man page just links to the regex crate docs, it doesn't generate a new man page for those & name that.
As for "filenames aren't even text" not being a useful model, to me text is a `&str` or `String` or `OsString`, filenames are a `Path` or `PathBuf`. We have different types for paths & strings because they represent different things, and have different valid contents. All I mean by that is the types are different, and the types you use for text shouldn't be the same as the types you use for paths.
I'd suggest engaging with this question, which I think you ignored:
> Are the contents of files text?
It is perhaps the most prescient of all. What is the OS interface for files? Does it tell you, "This is a UTF-8 encoded text file containing short human readable lines"? No, it does not. All you get is bytes, and if you're lucky, you can maybe infer something about the extension of the file's path (but this is only a convention).
How do you turn bytes into a `&str`? Do you think ripgrep converts an entire file to `&str` before searching it? Does ripgrep even do UTF-8 validation at all? No no no, it does not.
To be clear, there is no perfect answer here. You've got to do the best with what you've got. But the model I work with is, "treat file contents and file paths as text until you heuristically believe otherwise." But I work on Unix CLI tooling that needs to be fast. For most people, I would say, "validate file contents and file paths as text" is the right model to start with.
> but IMO it should be pointed out in the man page
Docs can always be improved, sure, but that is not what I'm trying to engage with you about. :-)
I'd say some files are text, some are not. And I agree that there's no good way to tell! I think ripgrep has a much harder job than fd, because at least fd can always know that all paths it's searching are valid paths for the OS in use.
I get it. I think you're right that they both have the same problem, but paths have a std type for handling them that, while file content's don't. As long as you're on an OS you can use std::path::Path (or PathBuf) for paths, and ensure they're valid. I suppose I should have said "Paths aren't Strings" or similar, they might be text but they might not be, and fundamentally the issue is that they're different data types. "Text" isn't universally defined.
You can't really just use `std::path::Path` though. Because it's largely opaque. How do you run a regex or a glob on a `std::path::Path`? Doing a UTF-8 check first is expensive at ripgrep's scale. So it just gets it to `&[u8]` as quickly as it can and treats it as if it were text. (These days you can use `OsStr::as_encoded_bytes`.)
`std::path::Path` isn't necessarily a better design. I mean, on some days, I like it. On other days, I wonder if it was a mistake because it creates so much ceremony. And in many of those cases, the ceremony is totally unwarranted.
And I'm saying this as someone who has been adjudicating Rust's standard library API since Rust 1.0.
What's happening here is that fd's `-e/--extension` flag requires that its parameter value be valid UTF-8. The only case where this doesn't work is if you want to filter by a file path whose extension is not valid UTF-8. Needless to say, I can't ever think of a case where you'd really want to do this.
But if you do, fd still supports it. You just can't use the `-e/--extension` convenience:
$ touch $'a.\xFF'
$ fd '.*\.(?-u:\xFF)'
a.�
That is, `fd` requires that the regex patterns you give it be valid UTF-8, but that doesn't mean the patterns themselves are limited to only matching valid UTF-8. You can disable Unicode mode and match raw bytes via escape sequences (that's what `(?-u:\xFF)` is doing).
So as a matter of fact, the sibling comments get the analysis wrong here. `fd` doesn't assume all of its file paths are valid UTF-8. As demonstrated above, it handles non-UTF-8 paths just fine. But some conveniences, like specifying a file extension, do require valid UTF-8.
Right because file names are not guaranteed to be UTF-8. That's the reason Rust has str and then OsStr. You may assume Rust str is valid UTF-8 (unless you have unsafe code tricking it) but you may not assume OsStr is valid UTF-8.
Here an invalid UTF-8 is passed via command line arguments. If it is desired to support this, the correct way is to use args_os https://doc.rust-lang.org/beta/std/env/fn.args_os.html which gives an iterator that yields OsString.
No, OsStr is for OSes that don't encode strings as UTF-8, e.g. Windows by default. Use Path or PathBuf for paths, they're not strings. Pretty much every OS has either some valid strings that aren't valid paths (e.g. the filename `CON` on Windows is a valid string but not valid in a path), some valid paths that aren't valid strings (e.g. UNIX allows any byte other than 0x00 in paths, and any byte other than 0x00 or `/` in filenames, with no restrictions to any text encoding), or both.
UTF-8 is common now, but it hasn't always been. Wanting support for other encoding schemes is a valid ask (though, I think the OP was needlessly rude about it).
But yeah I suppose you would need support for all the other foreign-language encodings that came in between -- UCS-2 for example.
But basically nobody does that. Glib (which drives all GTK apps' and various other apps file reading) doesn't support anything other than UTF8 filenames. At that point I'd consider the "migration" done and dusted.
The world is a lot more complicated & varied than you think :) I was digging around in some hard drives from 2004 just last weekend. At that time, lots of different encodings were common, especially internationally. Software archaelogy is a common hobby, it could be nice to be able to use a tool like this to search through old filesystems. "Not worth the effort" is definitely a valid response to the feature request, but that also doesn't mean there is absolutely no use for the feature.
I can definitely see a use case for supporting non-UTF-8 pathnames on disk (primarily for archaeological purposes).
In a UTF-8-path-only world, what I would do is have a mount option that says that the pathnames on disk are Latin-1 (so that \xff is mapped to U+00FF in UTF-8, which I'm too lazy to work its exact binary representation right now), and let the people doing archaeology on that write their own tools to remap the resulting mojibake pathnames into more readable ones. Not the cleanest solution, but there are ways to support non-UTF-8 disks even with UTF-8-only pathnames.
Oh yeah I can imagine the pain for drives from that era. I remember reading that sometimes you need the right "codebook" - what was the word - installed and stuff like that.
You do not have (or write programs for) filesystems that contain loads of ancient mp3 and wma files.
It is the bane of my existence, but many programs support all the Latin-1 and other file name encodings that are incompatible with UTF-8, so users expect _your_ programs to work too.
Now if you want me to actually _display_ them all correctly, tough turds...
Running a shell script went badly, generating a bunch of invalid files containing random data in their names, rather than one file containing random data.
You wish to find and delete them all, now that they've turned your home directory into a monstrosity.
*nix filenames are series of bytes, not UTF-8 (or anything else) strings. If a find replacement doesn't accept valid (parts of) filenames as input, it's a bit unfortunate.
If all you want to do is match against a sequence of bytes, sure. But when you want to start providing features like case-insensitivity, matching against file extensions, globbing, etc, then you have to declare what a given byte sequence actually represents, and that requires an encoding.
Your Turkish example doesn't work with `fd` because `fd` doesn't support specific locales or locale specific tailoring for case insensitive matching. It only supports what Unicode calls "simple case folding." It works for things far beyond English, as demonstrated above, but definitely misses some cases specific to particular locales.
TR18 used to have a Level 3[1] with the kind of locale-specific custom tailoring support found in GNU's implementation of POSIX locales, but it was so fraught that it was retracted completely some years ago.
fd is also a file & directory maintenance command by Shirai Takashi. (Package name: fdclone) I think it has been around for at least couple of decades.
I honestly never use find anymore, I just use ripgrep(rg). rg is so fast, that I don't worry about the ridiculously extra overhead of searching content. I can't remember the last time I used find.
I was just thinking yesterday that I'd appreciate a more modern alternative to find(1). Not a simplified one, though!
The original problem that led me to thinking about this, was that I wanted to do a search that would include all files below the current directory, except if they were within a specific subdirectory.
(Why not just `grep -v` for that directory? Because the subdir contained millions of files — in part, I was trying to avoid the time it would take find(1) just to do all the system calls required to list the directory!)
And yes, this is possible to specify in find(1). But no, it's not ergonomic:
find . -path ./excluded_dir -prune -o -type f -print
You can add parentheses, if you like. (But you don't have to. It seems find(1) has implicit Pratt parsing going on):
This incantation entirely changed my perspective on find(1) as a tool. Until I learned this, I had never understood exactly why find(1) nags you to put its arguments in a specific order.
But this makes "what find(1) does under the covers" very clear: it's assembling your expression into a little program that it executes in the context of each dirent it encounters on its depth-first traversal. This program has filtering instructions, like `-path`, and side-effect instructions, like `-print`. You can chain as many of each as you like, and nest them arbitrarily within (implicit or explicit) logical operators.
After some further experimentation, I learned that find's programs are expression-based; that every expression implicitly evaluates to either true or false; and that any false return-value anywhere in a sub-expression, is short-circuiting for the rest of that sub-expression. (You could think of every instruction as being chained together with an implicit &&.) This means that this won't print anything:
find . -false -print
In other words, my original expression above:
find . -path ./excluded_dir -prune -o -type f -print
...is actually this, when translated to a C-like syntax:
As soon as I realized this, I suddenly became very annoyed with find(1)'s actual syntax. Why is find(1) demanding that I write in this weird syntax with leading dashes, various backslash-escaped brackets, etc? Why can't I "script" find(1) on the command-line, the same way I'd "script" Ruby or Perl on the command-line? Or heck, given that this whole thing is an expression — why not Lisp?
I do understand why find(1) was designed the way it was — it's to allow for shell variable interpolation, and to rely on shell argument tokenization.
But it's pretty easy to work around this need — just push both of these concerns "to the edge" (i.e. outside the expression itself. Like SQL statement binding!)
• Support $VAR syntax for plain references to exported env-vars.
• Support positional binding syntax (?1 ?2 ?3) for positional arguments passed after the script.
• Support named binding syntax (?foo) for positional arguments after the script that match the pattern of a variable-assignment (ala make(1) arguments):
You might be interested in rawhide[1] or fselect[2]. (Note: I don't really use them myself, but they seem to offer something like what you're suggesting.)
Also, this is still a find-style syntax, but my bfs utility supports -exclude [3]. So you can write
bat, fd, hexyl, hyperfine
I'm going to take this moment to remind all of you well-paid engineers that if we each spread $10 a month sponsoring talented software makers like sharkdp the Internet would be a better place.
So many great little tools out there and we should try to support an ecosystem for them.