Here’s another example of “what’s old is new again” – remember how a long time ago, you interacted with a modem by giving it textual commands, and then it connected you to distant machines, which you also spoke to in text, and when you wanted to send and receive binary files, you had to encode those as text too?

Well, that still works, and the commands needed to encode/decode it are installed by default pretty much everywhere, so that means you can…

• Suppose there’s some system you connect to through a VPN and then two jump boxes. You’ve ssh’d all the way there, but were lazy and didn’t bother port-forwarding (if that’s even allowed), and now you need to get a copy of some config file. Instead of copy/pasting it a bit at a time, or trying to make your scrollback buffer and text wrapping cooperate (and still convert tabs to weird numbers of spaces…), you can:

on the sending side: cat file.conf | base64

Now you don’t have to worry about formatting at all*! Just copy all the base64 text as a block, and on the receive side: base64 -d > file.conf_from_remote

now paste the text, press enter, then ctrl+d when you’re done, and you have a binary-identical copy of the file on your local system, regardless of how many spaces, newlines, and messed up terminal wrapping you copied.

• The caveat: sometimes you’ll run into this on decode: “base64: invalid input”. In that case, try base64 -di as the decode command – for some weird reason, certain versions of the base64 utility can’t even decode their own input by default, because they decide to insert newlines on encode, but barf immediately on any non-base64 character on decode…including newlines. I have seen this behaviour primarily on old Gentoo boxes, Solaris, and ancient versions of CentOS and Red Hat.

• Doesn’t even have to be a remote system of course. I use this sometimes when I can’t be arsed to deal with sudo/chmod/chown when copying a file between sessions running as different restricted users, or across a chroot, container, VM, etc.

Next trick:
Suppose you’re editing a file locally and you want to copy a piece of a remote file, and it’s very important to exactly preserve the indenting and whitespace (because it’s python, yaml, or you’ve forgotten about “:set paste” in vim and internalised the notion that auto-indent is forever…but “set paste” doesn’t help you with tabs not surviving a terminal display anyway). You can do this:

shift+V to go to visual select line mode; select the block you want

type :! base64 <enter>

copy & paste the block into your other vim, then select the base64 text

type :! base64 -d <enter>

and there it is, in all its tabular/nonprinting/emoji/16-bit-big-endian-unicode-because-why-not glory. (You’ll want to undo the encode step on the source system, obviously.)

Don’t believe me that it’s 100% binary identical? Select the text blocks on both sides and check:

:! md5sum

(Incidentally, if the block of text you want is really small or your local one is very similar already, you can skip the base64 and just edit it manually and just use md5sum to confirm you got it right.)

If your file or block of text is longer than a screenful
Pipe it to gzip first:

cat file.txt | gzip -9 | base64

base64 -d | gunzip > file.txt_copy

(For very small inputs, gzip often produces slightly fewer bytes than xz and even zstd, plus it’s available practically everywhere.)

You can also scrunch down the base64 a little more by setting the line-width to unlimited (base64 -w 0), but be aware that:

• Some implementations are buggy when it comes to very long lines (the opposite problem of the earlier caveat).

• Even if the base64 command is OK with it, sometimes the terminal program isn’t.

• 4096 bytes per line is a common threshold at which something barfs.

• It can make the copy/pasting more error-prone, as it’s easier to miss a single character somewhere (and if you accidentally paste it in the wrong place, it makes more of a mess… on the other hand, at least your shell history will only have one bogus entry on accidental paste instead of 150. Ask me how many times I’ve seen “-bash: H4sIAAAAAAACAxXJQQ6AIAxE0b2nmJu49RoVxmgiLaFFw+2V3X/5m71IooiTUAakWNeAHaBGszpm: No such file or directory -bash: ztn1etic2Iki7r/ugczUKM68Lh893ENmSgAAAA==: No such file or directory” :P).

Important note for sysadmins and especially network people
I mentioned serial connections at the beginning of this. I cannot believe how many times I’ve see people laboriously copy a few lines at a time, paste them into their terminal window, wait (9600 8 N 1 only goes so fast, y’all…), copy a few more… and then cross their fingers and pray that no characters got lost, and none of the accidental extra whitespace will matter, when restoring a switch configuration.

The civilised way to do this is to be in shell mode on the switch instead of config mode (and if your switches don’t have a basic Linux-like shell, consider switching to some that do), and do a base64 copy/paste as described, and then compare checksums. Especially if gzip is available on the switch, this is much, much faster and more reliable, and then you can do a local “load config” and not have any terminal issues in config mode.

(Some may argue that transferring over tftp or some variant of DHCP-mediated auto-provision is “more civilised”, but 1, you’re in this situation because your network is buggered so that might not be an option, and 2, I bet if you held a race, the base64 person would be done long before the tftp person has even finished the “how the crap do I get this server listening again?! why is it not serving files?!” stage of cursing, never mind the “I fat-fingered a subnet mask” or “oh yeah, we block tftp at the firewall for this subnet now, don’t we?” stages of cursing.)

If your remote system is weird and doesn’t have a base64 command
Good chance it still does and it’s just part of something else. Hint: openssl has it built in (openssl base64 is equivalent to base64) if that’s available (e.g. Juniper switches I think). openssl md5 also works if you’re missing md5sum, but also try just md5, because it’s called that on some unixes (I want to say Juniper switches again? or Mac OS?).

du -h / --max-depth 5 | grep -P '^[0-9\.]+G' | sort -n -r

The output of the command will look something like this:

17G     /
8.2G    /home
4.8G    /usr
2.2G    /usr/local/cpanel
2.2G    /usr/local
1.3G    /run/log/journal/8936098ee7011e65401ba43c5f52e098
1.3G    /run/log/journal
1.3G    /run/log
1.3G    /run
1.1G    /var
1.1G    /backup

In my case, I ran this from a server that doesn’t have much installed, but you get the idea. By default this command will check the ‘/’ partition/directory (du -h /), iterate 5 levels deep (–max-depth 5), and only display directories that are at least 1GiB in size (grep -P ‘^[0-9.]+G’). You can change up the values to fine tune the output to your needs. For example, lets say you want to start at the current directory, show 2 levels deep, and only show directories that are at least 1MiB in size, but under 1GiB:

du -h . --max-depth 2 | grep -P '^[0-9\.]+M' | sort -n -r

In this case, I was within ‘/backup’, so the output would look like this:

555M    ./2020-12-13
514M    ./2020-12-13/accounts
503M    ./monthly/2020-12-01
503M    ./monthly
41M     ./2020-12-13/system
12M     ./.meta

This is a very useful command that I personally utilize just about every day. Let me know how useful this has been for you in the comments.