Matthew Bauer’s Blog

Owning Your Home’s Utilities

Sun, 09 Jun 2024 00:00:00 -0500

Ideally, utilities, such as water, electricity and gas, should be owned by its customers and not for-profit corporations. That would mean any money left over at the end of the year are passed on as future savings to ratepayers. In reality, this is not the case, and most Americans have at least one utility with for-profit ownership.

What’s interesting, however, is that these privately owned utilities are often publicly traded. So the everyday investor, with enough money, can actually purchase a chunk of their utilities and simulate the kind of customer ownership that publicly owned utilities have. The question is, what’s the right allocation? I’ve been thinking about this recently, and believe I have a full answer.

You want to buy just enough shares of the utility company so that the profit the company makes off of your usage equals the profit you make from your stake. If you have any more shares than that, then you’re taking some profits from other customers. If you have any less (or zero) shares, then an investor is capturing your profits.

So, my allocation formula looks like this:

Sorry, your browser does not support SVG. 1

Interestingly, earnings cancels out in this formula, so we don’t actually need to know about profitability, just revenue. We want just enough shares so that the revenue taken from us is equal to the revenue our shares represent. This can further be simplified to:

Sorry, your browser does not support SVG. 2

To determine optimal allocation, the two things we will need to calculate are personal spending and shares per revenue. Personal spending is not too hard to figure out, but number of shares per revenue is not a common metric you usually find in prospectuses - usually number of shares is in the denominator not the numerator. But there is a related metric we can use for this: Revenue Per Share (RPS). So the above formula can be rewritten as:

Sorry, your browser does not support SVG. 3

Since we cannot predict the future, we have to look backwards at these two metrics. For my own purposes, I am looking at 2023 numbers, which will hopefully even out some of the fluctuations in consumption that happens with seasonal changes.

1. Example

For this exercise, I’ve looked at my own utilities. I’ve omitted telephone & cable which I don’t use. I also excluded cellular which is not quite a monopoly where I am (U.S. cellular networks include AT&T, Dish, T-Mobile and Verizon).

Utility	Ownership
Electricity	Invester owned
Water	Publicly owned
Sewer	Publicly owned
Natural Gas	Invester owned
Trash	Invester owned
Internet	Invester owned

For where I live, water and sewer are publicly owned. All other utilities are owned by investors, and can be bought on the stock market. Here’s my spending for each of these utilities, as well as:

Utility	Stock symbol	My 2023 Spending
Electricity	NASDAQGS:EVRG	$1,374.58
Natural Gas	NYSE:OGS	$1,147.89
Trash	NYSE:GFL	$229.05
Internet	NASDAQ:GOOGL	$840

Some of this spending is rough estimates. Some percentage of spending in all of the above goes to sales tax, franchise fees, and other miscellaneous fees that may not actually reach the utility company. But, my opinion is a rough estimate is probably okay.

1.1. Utilities

1.1.1. Electricity

Evergy is the only utility of electricity in my area. I was able to pull financial data from its 2023 10-K:

Evergy 2023	(millions)
Revenue	$5,508.2
Net Income	$743.6
Number of shares	230

This gives RPS of $23.95/share. According to the above formula, I need to purchase 57.4 shares of Evergy. As of end of day June 3rd, 2023, Evergy traded at $52.29/share. So this would require a $3,001.45 investment.

Notably, this is looking at the profitability of the whole company. It’s possible that my specific home is more profitable than the average Evergy customer. For instance, a big cost for electricity is getting the power line to the home, which has already been done in my case. Or maybe some regions are more profitable than others. However, the same thing I think happens in public utilities - some houses are cheaper than others, yet the rates stay the same. So, I think looking at it from a whole company view is both simpler, and closely mirrors how public utilities actually run in practice.

1.1.2. Natural Gas

One Gas is the only utility of natural gas in my area. I was able to pull financial from its 2023 10-K:

ONE Gas 2023	(thousands)
Revenue	$2,371,990
Net Income	$231,232
Number of shares	55,600

This gives RPS of $42.66/share. According to the above formula, I need to purchase 26.91 shares of One Gas. As of end of day June 3rd, 2023, One Gas traded at $62.87/share. So this would require a $1,691.83 investment.

1.1.3. Trash

GFL is the trash service used in my neighborhood. Unlike the previous two companies, GFL is headquartered in Canada. However, it is still listed on the NYSE, so it can easily be purchased by Americans. The financial data is included in its 40-F instead of 10-K.

GFL 2023	(millions)
Revenue	$7,515.5
Net Income	$32.2
Number of shares	408

This gives RPS of $18.42/share. According to the above formula, I need to purchase 12.43 shares of GFL. As of end of day June 3rd, 2023, GFL traded at $34.73/share. So this would require a $431.69 investment.

1.1.4. Internet

I’ve saved Google Fiber, now called GFiber, for last because I think it is a special case in this. Alphabet (Google) is a huge company, and GFiber is only a small part of that. In financial statements, Alphabet has a broad category called "other bets" which includes GFiber as well as Waymo, Verily, and X. Other bets as a whole posted losses in the last couple of years. So most likely Alphabet is not making a profit on my $70/month I pay for GFiber. As a result, I think it doesn’t make sense to purchase shares in Alphabet.

1.2. My Asset Allocation

Here is what my asset allocation would look like:

	Investment	% of company	Allocation
Electricity	$3,001.45	0.00000025%	59%
Natural Gas	$1,691.83	0.00000048%	33%
Trash	$431.69	0.00000003%	8%
Total	$5,124.96		100%

While it may seem like a lot of money, this is actually much cheaper than I first thought it would cost. Most people don’t have this kind of money just sitting around, though.

Unlike most asset allocations, the percentages will change every year. For example, increased consumption of natural gas would mean higher allocation of natural gas. Increased price of natural gas, however, should be neutral on natural gas’s allocation because while revenue per share is going up, so is your personal spending.

The price of these shares will also change over time, but the allocation is really based on number of shares, not price. So, any price change shoudn’t require reallocation, but every time new revenue numbers come out, you may want to rebalance.

2. Open Questions

What’s the risk to this allocation?

I think by purchasing utilities, you are taking on risk that are held by the shareholders. So, in the event of bankruptcy, your equity would be wiped out. However, I’m skeptical that public utilities wouldn’t be bailed out in the event of insolvency. Also, publicly owned utilities already have this problem - ratepayers (and maybe also taxpayers) would be on the hook if the utility were insolvent, so it may not be much worse than that. I haven’t yet researched any cases of for-profit or pubicly owned utilities becoming insolvent.
What if you own your utilities through an index fund already?

I think you can think of your index fund as a separate pool of money from your direct ownership of utilities. Index funds are used to reduce overall risk, while this strategy is intended to capture the profits from your utility payments.
What if everyone did this?

It would be interesting if every ratepayer did this by buying all the shares of each utility. Would the utility become de facto publicly owned? Would company management care as much about rates as profits? All are fascinating questions that I don’t think we can answer.

Another crazy idea: what if publicly owned utilities gave each of their ratepayers shares: 1 share for $100 in spending. It would let ratepayers sell their shares if they needed money in return for higher realized rates rates. But it would be unclear how you would ensure future ratepayers also get equity. Perhaps it could work similar to how ESOP programs work in some workplaces.
Don’t state energy regulators represent consumers already?

State regulators do have a large role in regulation of energy and natural gas. They seem to mostly focus on pushing the utilities to keep rates low. But even the strictist state regulators will recognize a profit needs to be paid to shareholders each year.
What are some other applications of this?

You could take this allocation formula to other applications where you consume goods. For instance, if you regularly spend $100/month at Amazon, you could buy ($1200/Amazon RPS) shares in Amazon (where Amazon’s 2023 RPS is $55.78). This probably doesn’t work for Amazon in particular since they are a very diverse business and product sales like Amazon.com is probably less profitable than service sales like AWS. If you could purchase just the Amazon.com side of the business, you could perhaps do something like this. It might even be possible to do it yourself with some clever use of options trading. Perhaps, a business relying heavily on AWS would want to reap some of the profits they contribute to. There’s also a risk of being doubly exposed to risk in Amazon however - both your cloud platform and your balance sheet are effected if AWS becomes unprofitable.

Announcing Nixiosk

Mon, 23 Mar 2020 00:00:00 -0500

Today I’m announcing a project I’ve been working on for the last few weeks. I’m calling it Nixiosk which is kind of a smashing together of the words NixOS and Kiosk. The idea is to have an easy way to make locked down, declarative systems

My main application of this is my two Raspberry Pi systems that I own. Quite a few people have installed NixOS on these systems, but usually they are starting from some prebuilt image. A major goal of this project is to make it easy to build these images yourself. For this to work, I’ve had to make lots of changes to NixOS cross-compilation ecosystem, but the results seem to be very positive. I also want the system to be locked down so that no user can login directly on the machine. Instead, all administration is done on a remote machine, and deployed through SSH and Nix remote builders.

Right now, I have RetroArch (a frontend for a bunch of emulators) on my Raspberry Pi 4, and Epiphany (a web browser) on my Raspberry Pi 0. Both systems seem to be working pretty well.

GitHub: https://github.com/matthewbauer/nixiosk

1. Deploying

1.1. Install Nix

If you haven’t already, you need to install Nix. This can be done through the installer:

$ bash <(curl -L https://nixos.org/nix/install)

1.2. Cache

To speed things up, you should setup a binary cache for nixiosk. This can be done easily through Cachix. First, install Cachix:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install

Then, use the nixiosk cache:

$ cachix use nixiosk

1.3. Configuration

To make things simple, it just reads from an ad-hoc JSON file that describe the hardware plus some other customizations. It looks like this:

{
    "hostName": "nixiosk",
    "hardware": "raspberryPi4",
    "authorizedKeys": [],
    "program": {
        "package": "epiphany",
        "executable": "/bin/epiphany",
        "args": ["https://en.wikipedia.org/"]
    },
    "networks": {
        "my-router": "0000000000000000000000000000000000000000000000000000000000000000",
    },
    "locale": {
        "timeZone": "America/New_York",
        "regDom": "US",
        "lang": "en_US.UTF-8"
    },
    "localSystem": {
        "system": "x86_64-linux",
        "sshUser": "me",
        "hostName": "my-laptop-host",
    }
}

Here’s a basic idea of what each of these fields do:

hostName: Name of the host to use. If mDNS is configured on your network, this can be used to identify the IP address of the device via “.local”.
hardware: A string describing what hardware we are using. Valid values currently are “raspberryPi0”, “raspberryPi1”, “raspberryPi2”, “raspberryPi3”, “raspberryPi4”.
authorizedKeys: A list of SSH public keys that are authorized to make changes to your device. Note this is required because no passwords will be set for this system.
program: What to do in the kiosk. This should be a Nixpkgs attribute (package), an executable in that package, and a list of args.
networks: This is a name/value pairing of SSIDs to PSK passphrases. This can be found with the wpa_passphrase(8) command from wpa_supplicant.
locale: This provides some information of what localizations to use. You can set regulation domain, language, time zone via “regDom”, “lang”, and “timeZone”. If unspecified, defaults to US / English / New York.
localSystem: Information on system to use for remote builder. Optional.

1.4. Initial deployment

The deployment is pretty easy provided you have Nix installed. Here are some steps:

$ git clone https://github.com/matthewbauer/nixiosk.git
$ cd nixiosk/
$ cp nixiosk.json.sample nixiosk.json

Now you need to make some changes to nixiosk.json to reflect what you want your system to do. The important ones are ‘authorizedKeys’ and ‘networks’ so that your systems can startup and you can connect to it.

If you have an SSH key setup, you can get its value with:

$ cat $HOME/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC050iPG8ckY/dj2O3ol20G2lTdr7ERFz4LD3R4yqoT5W0THjNFdCqavvduCIAtF1Xx/OmTISblnGKf10rYLNzDdyMMFy7tUSiC7/T37EW0s+EFGhS9yOcjCVvHYwgnGZCF4ec33toE8Htq2UKBVgtE0PMwPAyCGYhFxFLYN8J8/xnMNGqNE6iTGbK5qb4yg3rwyrKMXLNGVNsPVcMfdyk3xqUilDp4U7HHQpqX0wKrUvrBZ87LnO9z3X/QIRVQhS5GqnIjRYe4L9yxZtTjW5HdwIq1jcvZc/1Uu7bkMh3gkCwbrpmudSGpdUlyEreaHOJf3XH4psr6IMGVJvxnGiV9 mbauer@dellbook

which will give you a line for “authorizedKeys” like:

"authorizedKeys": ["ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC050iPG8ckY/dj2O3ol20G2lTdr7ERFz4LD3R4yqoT5W0THjNFdCqavvduCIAtF1Xx/OmTISblnGKf10rYLNzDdyMMFy7tUSiC7/T37EW0s+EFGhS9yOcjCVvHYwgnGZCF4ec33toE8Htq2UKBVgtE0PMwPAyCGYhFxFLYN8J8/xnMNGqNE6iTGbK5qb4yg3rwyrKMXLNGVNsPVcMfdyk3xqUilDp4U7HHQpqX0wKrUvrBZ87LnO9z3X/QIRVQhS5GqnIjRYe4L9yxZtTjW5HdwIq1jcvZc/1Uu7bkMh3gkCwbrpmudSGpdUlyEreaHOJf3XH4psr6IMGVJvxnGiV9 mbauer@dellbook"],

and you can get a PSK value for your WiFi network with:

$ nix run nixpkgs.wpa_supplicant -c wpa_passphrase my-network
network={
        ssid="my-network"
        #psk="abcdefgh"
        psk=17e76a6490ac112dbeba996caa7cd1387c6ebf6ce721ef704f92b681bb2e9000
}

so your .json file looks like:

"networks": {
  "my-network": "17e76a6490ac112dbeba996caa7cd1387c6ebf6ce721ef704f92b681bb2e9000",
},

Now, after inserting your Raspberry Pi SD card into the primary slot, you can deploy to it with:

$ ./deploy.sh /dev/mmcblk0

You can now eject your SD card and insert it into your Raspberry Pi. It will boot immediately to an Epiphany browser, loading en.wikipedia.org.

Troubleshooting steps can be found in the README.

1.5. Redeployments

You can pretty easily make changes to a running system given you have SSH access. This is as easy as cloning the running config:

$ git clone ssh://root@nixiosk.local/etc/nixos/configuration.git nixiosk-configuration
$ cd nixiosk-configuration

Then, make some changes in your repo. After your done, you can just run ‘git push’ to redeploy.

$ git add .
$ git commit
$ git push

You’ll see the NixOS switch-to-configuration log in your command output. If all is successful, the system should immediately reflect your changes. If not, the output of Git should explain what went wrong.

Note, that some versions of the Raspberry Pi like the 0 and the 1 are not big enough to redeploy the whole system. You will probably need to setup remote builders. This is described in the README.

2. Technology

Here are some of the pieces that make the Kiosk system possible:

Cage / Wayland: Cage is a Wayland compositor that allows only one application to display at a time. This makes the system a true Kiosk.
NixOS - A Linux distro built on top of functional package management.
Basalt: A tool to manage NixOS directly from Git. This allows doing push-to-deploy directly to NixOS.
Plymouth: Nice graphical boot animations. Right now, it uses the NixOS logo but in the future this should be configurable so that you can include your own branding.
OpenSSH: Since no direct login is available, SSH is required for remote administration.
Avahi: Configures mDNS registration for the system, allowing you to remember host names instead of IP addresses.

I would also like to include some more tools to make administration easier:

ddclient / miniupnp: Allow registering external IP address with a DNS provider. This would enable administration outside of the device’s immediate network.

3. Project

You can try it out right now if you have an Raspberry Pi system. Other hardware is probably not too hard, but may require tweaking. The project page is available at https://github.com/matthewbauer/nixiosk and issues and pull requests are welcomed.

All the versions with Nix

Wed, 21 Aug 2019 00:00:00 -0500

1. Background

In Channel Changing with Nix, I described how to move between channels in Nix expressions. This provides an easy way to work with multiple versions of Nixpkgs. I was reminded of this post after seeing a comment by jolmg on Hacker News. The comment suggested we should have a way to use multiple versions of packages seamlessly together. It suggested that we should use commits to differentiate versions, but I suggested that stable channels would work much better.

So, as a follow up to my Channel Changing post, I want to show how you can use a quick Nix mockup to accomplish this. Like the previous post, it will come with a Nix snippet that you can try out yourself.

2. Code

So, what follows is some code that I wrote up that lets you find package versions in an easier way. It is also available for download at https://matthewbauer.us/generate-versions.nix.

{ channels ? [ "19.03" "18.09" "18.03" "17.09" "17.03"
               "16.09" "16.03" "15.09" "14.12" "14.04" "13.10" ]
, attrs ? builtins.attrNames (import  {})
, system ? builtins.currentSystem
, args ? { inherit system; }
}: let

  getSet = channel:
    (import (builtins.fetchTarball "channel:nixos-${channel}") args).pkgs;

  getPkg = name: channel: let
    pkgs = getSet channel;
    pkg = pkgs.${name};
    version = (builtins.parseDrvName pkg.name).version;
  in if builtins.hasAttr name pkgs && pkg ? name then {
    name = version;
    value = pkg;
  } else null;

in builtins.listToAttrs (map (name: {
  inherit name;
  value = builtins.listToAttrs
    (builtins.filter (x: x != null)
      (map (getPkg name) channels));
}) attrs)

This Nix expression generates an index of each package from all 11 releases of Nixpkgs that have occurred since October 2010. For every package, each version that came with a release is included and put into a map. The map uses the version as a key and the package as its value, preferring the newer release when versions conflict.

This is all done lazily, because that’s how Nix works. Still, it will take a little while at first to evaluate because we need to parse all 11 releases! Remarkably, this expression uses only Nix builtins, and requires no special library function.

3. Usage

Working with this Nix expression is extremely interesting, and I’ve included some examples of how to work with it. They should all be usable on a Linux machine (or maybe macOS) with Nix installed.

3.1. Query package versions

You can query what package versions are available through Nix’s builtins.attrNames function. For example,

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs)"
[ "24.3" "24.4" "24.5" "25.3" "26.1" ]

This shows us that there are 5 versions of Emacs. This is kind of interesting because it means that there were at least 6 duplicate versions of Emacs between our release channels. Unfortunately, a few versions of Emacs are notably missing including Emacs 25.1 and Emacs 25.2. Emacs 24.2 was released almost a year before the first stable Nixpkgs release! As time goes on, we should collect more of these releases.

3.2. Running an old version

As shown above, there are 5 versions of Emacs available to us. We can run Emacs 24.3 with a fairly short command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"24.3\"" -c emacs

LC_ALL=C is needed on Linux to avoid the old Glibc trying to load the newer, incompatible locales that may be included with your system. This is an unfortunate problem with Glibc including breaking changes between releases. It also makes me want use to switch to Musl some time soon! I’ve also noticed some incompatibilities with GTK icons that appear to come from the gdk-pixbuf module. More investigation is needed on why this is the case.

This will not work on macOS because we did not have Emacs working on macOS back then! macOS users can try Emacs 25.3. This looks very similar to the above:

$ nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"25.3\"" -c emacs

3.3. Firefox

Another example using Firefox is pretty neat. The code is very similar to Emacs:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox)"
[ "25.0.1" "34.0.5" "39.0.3" "45.0" "48.0.2" "51.0.1" "55.0.3" "59.0.2" "63.0.3" "66.0.3" "68.0.2" ]

We get all 11 releases with unique Firefox versions this time.

You can run Firefox 25.0.1 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox.\"25.0.1\"" -c firefox

Amazing how notably Firefox has changed since then!

3.4. Blender

Another example using Blender. The code is very similar to the two above:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender)"
[ "2.67" "2.70" "2.72b" "2.75a" "2.77a" "2.78c" "2.79" "2.79a" "2.79b" ]

You can run Blender 2.67 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender.\"26.7\"" -c blender

4. Rationale

The reason that channels work better than commits is because every commit in Nixpkgs is not guaranteed to work on its own. Some may be missing security patches, configuration changes, or worse may just not work with other versions of packages. In addition, there are just too many commits to work with effectively. On the other hand, Nixpkgs release stable channels every 6 months, and we have a long vetting process of ensuring the stabilized channel works well.

The main drawback the 6-month channels have is that we don’t have every version released of package. If the version you want is missing in a release, you are out of luck. But, the 6-month window tends to pick up a lot of packages and we end up with almost every major version of popular software. My philosophy is not all releases are worth keeping. Some contain critical security flaws, contain major bugs, and might not work well with other software. The 6-month window is good enough for me. Perhaps in the future we can increase Nixpkgs release cadence to 3-month or 1-month, but the maintainers are not quite ready for that yet.

5. Conclusion

This has hopefully shown how Nix’s functional dependency model makes it very easy to switch between versions of packages. This is builtin to Nix, but you need some scripts to really use this well. Our 6-month release window is an arbitrary choice, but tends to pick up a lot of useful versions in the mean time.

Nixpkgs macOS Stdenv Updates

Mon, 06 May 2019 00:00:00 -0500

Over the past couple of months, I have been working on updating the macOS stdenv in Nixpkgs. This has significant impact on users of Nix/Nixpkgs on macOS. So, I want to explain what’s being updated, what the benefits are, and how we can minimize breakages.

1. macOS/Darwin stdenv changes

First, to summarize the changes that impact stdenv and the Darwin infrastructure. The PR is available at NixOS/nixpkgs PR #56744. This update has been in the works for the last few months, and is currently in the staging-next branch, waiting to be merged in NixOS/nixpkgs PR #60491. It should land on master and nixpkgs-unstable in the next few days. The main highlights are —

Change default LLVM toolchain from 5 to 7. LLVM 5 stdenv is still available through llvmPackages_5.stdenv attribute path.
Upgrade Apple SDK from 10.10 to 10.12.
Update libSystem symbols from 10.10 (XNU 3789.1.32) to 10.12 (XNU 3789.1.32).
Removed old patches to support old stdenv in Qt 5 and elsewhere.

These macOS SDK upgrades are equivalent to setting -mmacosx-version-min to 10.12 in XCode. As a result, we will break compatibility with macOS before 10.12.

2. Why do we need to set a minimum macOS version?

Without knowing internals of Nixpkgs, it might not be clear why we need to set a minimum macOS version. For instance with Linux, we are able to support any Linux kernel in Nixpkgs without any problem. The answer to this requires some understanding of how the kernel and userspace function.

Nixpkgs is able to support multiple Linux kernels because we can use multiple Libc’s at one time. For any executable, a Nix closure will include both its own Libc and the dynamic linker in its closure. This works in Linux where multiple Libc’s can be used, but not on macOS where only one Libc is available.

In short, Linux and macOS deal with compatibility between built binaries in different ways. They represent two opposite ends in how Unix-like kernels maintain compatibility with their userspace binaries.

2.1. Linux syscall compatibility

The kernel is responsible for managing core operating system functions such as start-up, memory management, device abstractions, and process isolation. For it to function, the kernel needs to interact with the rest of the operating system which is collectively referred to as “userspace”. Executables in userspace use “syscalls” to tell the kernel what to do. These syscalls are very low-level and usually not called directly by a process. Instead, an abstraction layer is provided by the standard C library, or Libc.

Linux is unique among operating systems due to the fact that the Kernel and Libc are developed independently. Linux is maintained by creator Linus Torvalds and a community of contributors. Glibc, the most popular Libc for Linux, is maintained by the GNU project. As a result, Linux has a strong separation between Syscalls and Libc.

Linux does not tie itself to any specific Libc. Even though Glibc is used in almost all distros, many alternatives are available. For instance, Musl provides a more lightweight version of Glibc, while Bionic is the Libc used in the Android operating system. In addition, multiple versions of each of these Libc’s can be used on any one kernel, even at the same time. This can become very common when using multiple Nixpkgs versions at one time.

To accomplish this, Linux provides a stable list of syscalls that it has maintained across many versions. This is specified for i386 at arch/x86/entry/syscalls/syscall_32.tbl in the kernel tree. The syscalls specified here are the interface through which the Libc communicates with the kernel. As a result, applications built in 1992 can run on a modern kernel, provided it comes with copies of all its libraries¹.

2.2. macOS Libc compatibility

The macOS Libc is called libSystem. It is available on all macOS systems at /usr/lib/libSystem.B.dylib. This library is the main interface that binary compatibility is maintained in macOS. Unlike Linux, macOS maintains a stable interface in libSystem that all executables are expected to link to. This interface is guaranteed by Apple to be stable between versions.

In Nixpkgs, we maintain this compatibility through a list of symbols that are exported by libSystem. This is a simple text list and is available for viewing at NixOS/nixpkgs/pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_c_symbols. The symbol list is created by listing symbols (nm) on the minimum macOS version that we support (for my PR, 10.12). We do some linking tricks to ensure that everything that we build in Nixpkgs only contains those symbols. This means that we can reproducibly build on newer versions of macOS, while maintaining compatibility with older macOS versions. Unfortunately, newer symbols introduced in later versions cannot be used even on systems that have those symbols.

A side effect of macOS design, is that fully static executables are not supported in macOS as they are on Linux. Without a stable syscall interface, there is nothing to provide compatibility between versions. As a result, Apple does not support this type of linking².

There is no mandated reason why we need to use libSystem directly. In fact, some languages like Go have attempted to instead use the syscall interface directly. There is no reason why this couldn’t work, however, upgrades between versions will almost certainly break binaries. Go eventually abandoned this scheme in time for Go 1.12 (proposed by Nixpkgs macOS contributor copumpkin!)

2.3. Others

Some other examples may be useful. They mostly fall on one side or the other of the Syscall / Libc divide —

FreeBSD - breaks syscall compatibility between major releases, should use Libc for longterm binary compatibility.
OpenBSD - similarly, changes syscall interface, perhaps even more often than FreeBSD³.
NetBSD - apparently has maintained syscall compatibility since 1992. ⁴
Windows, Solaris, Fuchsia - I cannot find any information on these and how they handle binary compatibility.

2.4. LLVM triple

As a side note, this difference can be clearly seen in how we specify target systems. The LLVM triple is a 3 or 4-part string specifying what we want to build for. The parts of the triple correspond to:

---

— the CPU architecture that we are building for. Examples include x86_64, aarch64, armv7l, etc.
— an arbitrary string specifying the vendor for the toolchain. In Nixpkgs, this should always be unknown.
— the kernel to build for (linux).
— the kernel ABI to use. On Linux, this corresponds to the Libc we are using (gnu for Glibc, musl for Musl, android for Bionic).

When building for Linux, we can build for any version of Linux at one time. No version information is required. In addition, we must specify what “ABI” we want to use. In Nix, this is not very important because the Libc is provided by the closure. In fact, Nix has its own version of the LLVM triple called a Nix system tuple that omits the portion altogether! It corresponds to - from the LLVM triple.

In comparison, when building for BSDs, we must specify which version of the kernel we are building for. In addition, we leave off the portion, because there is only one Libc available for these platforms. They are even included in the same tree as the kernel. Examples of BSD triples include,

aarch64-apple-darwin16.0.0
x86_64-unknown-freebsd12.0
i386-unknown-openbsd5.8
armv7l-unknown-netbsd7.99

3. Compatibility table

Looking through the old versions, I’ve compiled a list of what I think are the corresponding macOS versions for each Nixpkgs release. As you can see, we try to support at least 3 previous macOS releases. This also happens to be about what Apple supports through security updates⁵.

Nixpkgs release	macOS version
19.09	10.12, 10.13, 10.14, 10.15?
19.03	10.11⁶, 10.12, 10.13, 10.14
18.09	10.11, 10.12, 10.13, 10.14
18.03	10.11, 10.12, 10.13, 10.14
17.09	10.10, 10.11, 10.12, 10.13
17.03	10.10, 10.11, 10.12
16.09	10.10, 10.11, 10.12
16.03	10.9?, 10.10, 10.11, 10.12

We know that some users are stuck on older versions of macOS due to reasons outside of their control. As a result, we will try to support the 19.03 branch for a little bit longer than is usually done. If your organization uses 10.11, it might be a good idea to update to a newer version along with your update to Nixpkgs 19.09.

4. Conclusion

My main goal has been to show better how Nixpkgs and macOS system interact. I got a little bit sidetracked exploring differences in binary compatibility between different operating systems. But, this should help users to better understand the differences in how macOS and Linux works in relation to Nixpkgs.

Footnotes:

It would be interesting to test this in practice. Finding a Libc that would work might be the hardest part. Even better if we could use Nix’s closures!

StackOverflow question at https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x.

According to the_why_of_y on Hacker News, https://news.ycombinator.com/item?id=14011662

⁴

Some more details on NetBSD’s ABI are available at http://www.jp.netbsd.org/gallery/presentations/joerg/asiabsdcon2016/asiabsdcon2016.pdf

⁵

macOS updates come out about every year and Apple offers about 3 months support. More information is available at https://apple.stackexchange.com/questions/47664/what-is-apples-policy-for-supporting-security-updates-on-older-versions-of-os-x.

⁶

There is an issue with building on 10.11 with the new swift-corelibs derivation. As a result, you need to use prebuilt version to avoid this issue.

Static Nix: a command-line swiss army knife

Sun, 24 Feb 2019 00:00:00 -0600

Nix is an extremely useful package manager. But, not all systems have it installed. Without root priveleges, you cannot create the /nix directory required for it to work.

With static linking, and some new features added in Nix 2.0, you can fairly easily use the Nix package manager in these unpriveleged context¹. To make this even easier, I am publishing prebuilt a x86_64 binary on my personal website. It will reside permanently at https://matthewbauer.us/nix (5M download).

1. Trying it out

You can use it like this,

$ curl https://matthewbauer.us/nix | sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable hello -c hello
Hello World!

You can use any package provided by Nixpkgs (using the attribute name). This gives you a swiss army knife of command line tools. I have compiled some cool commands to try out. There examples of various tools, games and demos that you can use through Nix, without installing anything! Everything is put into temporary directories².

1.1. Dev tools

$ nix=$(mktemp); \
  curl https://matthewbauer.us/nix > $nix && \
  chmod +x $nix && \
  $nix run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  bashInteractive curl git htop imagemagick file findutils jq nix openssh pandoc

1.2. Emacs

$ nix=$(mktemp); \
  curl https://matthewbauer.us/nix > $nix && \
  chmod +x $nix && \
  $nix run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  emacs -c emacs

1.3. File manager

$ nix=$(mktemp); \
  curl https://matthewbauer.us/nix > $nix && \
  chmod +x $nix && \
  $nix run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  ranger -c ranger

1.4. Fire

$ curl https://matthewbauer.us/nix | \
  sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  aalib -c aafire

1.5. Fortune

$ curl https://matthewbauer.us/nix | \
  sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  bash cowsay fortune -c sh -c 'cowsay $(fortune)'

1.6. Nethack

$ nix=$(mktemp); \
  curl https://matthewbauer.us/nix > $nix && \
  chmod +x $nix && \
  $nix run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  nethack -c nethack

1.7. Weather

$ curl https://matthewbauer.us/nix | \
  sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  bash curl cowsay -c sh -c 'cowsay $(curl wttr.in/?format=3)'

1.8. World map

$ curl https://matthewbauer.us/nix | \
  sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  bash coreutils curl libcaca ncurses -c bash -c \
  'img=$(mktemp ${TMPDIR:-/tmp}/XXX.jpg); \
  curl -k https://www.cia.gov/library/publications/the-world-factbook/attachments/images/large/world-physical.jpg > $img \
  && img2txt -W $(tput cols) -f utf8 $img'

1.9. Youtube

$ curl https://matthewbauer.us/nix | \
  sh -s run --store $HOME/.cache/nix/store -f channel:nixpkgs-unstable \
  bash youtube-dl mplayer -c sh -c \
  'mplayer -vo caca $(youtube-dl --no-check-certificate -g https://www.youtube.com/watch?v=dQw4w9WgXcQ)'

1.10. And more…

Lots more cool things are possible. Look through the packages provided by Nixpkgs if you need inspiration.

2. Avoid installing and extracting each time

This method of using Nix has some upfront cost. This is because https://matthewbauer.us/nix must be downloaded each time and the embedded .tar.gz file extracted. If you want Nix to stay around permanently, you have to follow a few tricks. Total install size is about 11M. Using this method, you will reduce startup and keep Nix in your path at each login.

I have two ways of doing this. One the “easy” way is just running this script.

$ curl https://matthewbauer.us/nix.sh | sh

The other is the "safe" way and involves running some commands in order. These are the same commands run by the script, but this lets you audit everything being done line by line.

$ t=$(mktemp -d)
$ curl https://matthewbauer.us/nix > $t/nix.sh
$ (cd $t && sh nix.sh --extract)
$ mkdir -p $HOME/bin/ $HOME/share/nix/corepkgs/
$ mv $t/dat/nix $HOME/bin/
$ mv $t/dat/share/nix/corepkgs/* $HOME/share/nix/corepkgs/
$ echo export 'PATH=$HOME/bin:$PATH' >> $HOME/.profile
$ echo export 'NIX_DATA_DIR=$HOME/share' >> $HOME/.profile
$ source $HOME/.profile
$ rm -rf $t

You can now run the Nix commands above as you need to, and it will be available on each login. Remember to always add the arguments -f channel:nixpkgs-unstable and --store $HOME/.cache/nix/store, otherwise Nix will be confused on how to handle the missing /nix/store and other environment variables.

3. Build it yourself

This is certainly a security vulnerability so you may want to build static Nix for youself from my pull request. Of course you can’t build static Nix without Nix, so this would need to be done from a system that has Nix installed. You can build it yourself, provided you have git and nix installed, like this,

$ git clone https://github.com/matthewbauer/nixpkgs.git
$ cd nixpkgs
$ git checkout static-nix
$ nix-build -A pkgsStatic.nix

Then, copy it to your machine without Nix installed (provided you have ssh installed), like this,

$ scp ./result/bin/nix your-machine:
$ ssh your-machine
$ ./nix ...

Footnotes:

Note that you will need to be able to set up a private namespace. This is enabled by default on Linux, but some distros have specifically disabled it. See this issue for more discussion.

While ideally we would not need temporary directories at all, some of these commands require it. This is because they check whether they are in a pipe and refuse to run if so. Your temporary directory should be cleaned each time your reboot anyway. The Nix packages will be installed in $HOME/.cache/nix/store but they can be removed at any time.

Call for proofreaders and beta testers for 19.03

Fri, 08 Feb 2019 00:00:00 -0600

This was originally published on Discourse. I am putting it here for posterity reasons.

We get lots of contributors in Nixpkgs and NixOS who modify our source code. They are the most common type of contribution we receive. But, there is actually a great need for other types of contributions that don't involve programming at all! For the benefit of new users, I am going to outline how you can easily contribute to the community and help make 19.03 the best NixOS release yet.

1. Proofreading

We have two different manuals in the NixOS/nixpkgs repo. One is for Nixpkgs, the set of all software. And the other is for NixOS, our Linux distro. Proofreading these manuals is important in helping new users learn about how our software works.

When you find an issue, you can do one of two things. The first and most encouraged is to open a PR on GitHub fixing the documentation. Both manuals are written in docbook. You can see the source for each here:

GitHub allows you to edit these files directly on the web. You can also always use your own Git client. For reference on writing in DocBook, I recommend reading through docbook.rocks.

An alternative if you are unable to fix the documentation yourself is to open an issue. We use the same issue tracker includes any issues with Nixpkgs/NixOS and can be accessed through GitHub Issues. Please be sure to provide a link to where in the manual the issue is as well as what is incorrect or otherwise confusing.

2. Beta testing

An alternative to proofreading is beta testing. There are a number of ways to do this, but I would suggest using VirtualBox. Some information on installing VirtualBox can be found online, but you should just need to set these NixOS options:

virtualisation.virtualbox.host.enable = true;

and add your user to the vboxusers group:

users.users..extraGroups = [ "vboxusers" ];

then rebuild your NixOS machine (sudo nixos-rebuild switch), and run this command to start virtualbox:

VirtualBox

Other distros have their own ways of installing VirtualBox, see Download VirtualBox for more info.

You can download an unstable NixOS .ova file directly here. (WARNING: this will be a large file, a little below 1GB).

nixos-19.03pre-x86₆₄-linux.ova

Once downloaded, you can import this .ova file directly into VirtualBox using "File" -> "Import Appliance…". Select the .ova file downloaded from above and click through a series of Next dialogs, using the provided defaults. After this, you can boot your NixOS machine by selecting it from the list on the left and clicking "Start".

The next step is to just play around with the NixOS machine and try to break it! You can report any issues you find on the GitHub Issues tracker. We use the same issue tracker for both NixOS and Nixpkgs. Just try to make your issues as easy to reproduce as possible. Be specific on where the problem is and how someone else could recreate the problem for themselves.

Subjective ranking of build systems

Mon, 26 Nov 2018 00:00:00 -0600

1. My subjective ranking of build systems

Very few of us are happy with our choices of build systems. There are a lot out there and none feel quite right to many people. I wanted to offer my personal opinions on build systems. Every build system is “bad” in its own way. But some are much worse than others.

As a maintainer for Nixpkgs, we have to deal with . I’ve avoided build systems that are language-specific. Those build systems are usually the only choice for your language, so ranking them will inevitably include opinions on the language itself. So, I’ve included in this list only language neutral build systems. In addition, I’ve filtered out any build systems that are not included in Nixpkgs. This perspective is going to prioritize features that make your project easiest to package in cross-platform ways. It’s very subjective, so I only speak for myself here.

I separate two kinds of software used for packages. One is the “meta” build system that provides an abstract interface to create build rules. The other is the build runner that will run the rules. Most meta build systems support targeting multiple backends.

1.1. What makes a good build system?

Some criteria I have for these build systems.

Good defaults builtin. By default, packages should support specifying “prefix” and “destination directory”.
Works with widely available software. Being able to generate Makefiles is a big bonus. Everyone has access to make - not everyone has Ninja. This is often needed for bootstrapping.
Supports cross compilation concepts. A good separation between buildtime and runtime is a must have! In addition you should be able to set build, host, and target from the command line. This makes things much easier for packaging and bootstrapping.
Detection of dependencies reuses existing solutions. Pkgconfig provides an easy way to detect absolute directories. No need to reinvent the wheel here.
The less dependencies the better! Requiring a Java or Python runtime means it takes longer to rebuild the world. They introduce bottlenecks where every package needs to wait for these runtimes to be built before we can start running things in parallel.

1.2. Ranking of meta build systems from bad to worse

GNU Autotools (Automake & autoconf)
Meson
CMake
gyp
qmake
imake
premake

GNU Autotools comes in at the top. It has the best support for cross compilation of any meta build system. It has been around for a while and means that the classic “./configure && make && make install” work. Because the configure script is just a simple bash script, packages don’t have to depend directly on GNU Autotools at build time. This is a big plus in bootstrapping software. I think Meson has made a lot of progress in improving its cross compilation support. It’s not quite there in my opinion, as it requires you to create cross tool files instead of using command line arguments.

1.3. Ranking of build runners from bad to worse

GNU Make
Ninja
Bazel
boost.build
SCons

GNU Make is still the top choice in my personal opinion. It has been around for a while, but Makefiles are widely understood and GNU Make is included everywhere. In addition, the Makefile build rule format is easy to parallelize. Ninja still requires Python to build itself. This adds to the Nixpkgs bottleneck because Python is not included in the bootstrap tools. While there are some speedups in Ninja, they don’t appear to be significant enough to be worth switching at this time. At the same time, Ninja is still a perfectly good choice if you value performance over legacy support.

1.4. Conclusion

In Nixpkgs, we have made an attempt to support whatever build system you are using. But, some are definitely better than others.

My main goal here is to try to get software authors to think more critically about what build system they are using. In my opinion, it is better to use well known software over more obscure systems. These shouldn’t be taken as a universal truth. Everyone has their own wants and needs. But, if your build system comes in at the bottom of this list, you might want to consider switching to something else!

Beginner’s guide to cross compilation in Nixpkgs

Wed, 25 Jul 2018 00:00:00 -0500

1. What is cross compilation?

First, compilation refers to converting human-readable source code into computer-readable object code. Usually the computer you are building the code for is the same as the computer you are running on¹. In cross compilation, however, that is not the case. We can build code for any computer that our compiler supports!

Cross-compilation is not a new idea at all. GCC and Autoconf are ancient tools that we use internally in Nixpkgs. But, getting those ideas to work well with Nix’s functional dependency model has taken years and years of work from the Nix community. We are finally to the point where an end user can easily start cross compiling things themselves.

2. Unstable channel

If you do not have Nix installed, you can install it available at NixOS.org. The rest of the guide will assume that Nix is already installed.

Much work has gone into bringing cross compilation support to Nixpkgs. While Nixpkgs has had some support for cross compiling for awhile, recent changes have made cross compilation much easier and more elegant. These changes will be required for this guide. We plan to have a stable version of this ready for 18.09. Before that though, you will need to use the unstable channel. Things can be built with the unstable channel fairly easily with Nix 2.0. For instance, to build the hello program,

nix build -f channel:nixos-unstable hello

The rest of the guide will use nixos-unstable as the channel. However, once 18.09 is released, you should be able to also use the stable channel.

3. Building things

One of the important principles of cross compilation in Nixpkgs is handling native and cross compilation identically. This means that it should be possible to cross-compile any package in Nixpkgs with little to no modification at all. If your derivation specifies its dependencies correctly, Nix/Nixpkgs can figure out how to build it.

So now it’s time to show what we can do with Nixpkgs cross compilation framework². I’ve compiled a short list of cross package sets along with their corresponding attribute names.

Raspberry Pi (pkgsCross.raspberryPi)
x86_64 Musl (pkgsCross.musl64)
Android (pkgsCross.aarch64-android-prebuilt)
iPhone (pkgsCross.iphone64)
Windows (pkgsCross.mingwW64)

So, if you are familiar with Nixpkgs, you would know that if you wanted to build Emacs for your native computer you can just run,

$ nix build -f channel:nixos-unstable pkgs.emacs

Likewise, if you wanted to build Emacs for a Raspberry Pi, you can just run,

$ nix build -f channel:nixos-unstable pkgsCross.raspberryPi.emacs

The built package will be in the same ARM machine code used by the Raspberry Pi. The important thing to notice here is that we have the power to build in any package in Nixpkgs for any of the platforms listed above. Of course, many of these will have issues due to not being portable, but with time we can make both Nixpkgs & the free software world better at handling cross compilation. Any of the software listed in ‘nix search’ should be possible to cross compile through the pkgsCross attribute.

Some more examples of things that I have worked on,

Windows

$ nix build -f channel:nixos-unstable pkgsCross.mingw32.hello
$ nix run -f channel:nixos-unstable wine -c ./result/bin/hello.exe
Hello, world!

Android

$ nix build -f channel:nixos-unstable \
      pkgsCross.aarch64-android-prebuilt.curl

iPhone³

$ nix build -f channel:nixos-unstable \
      pkgsCross.iphone64.haskell.packages.jq

Notice that the pkgsCross attribute is just sugar to a more powerful & composable interface to Nixpkgs. This can be specified from the command line with,

$ nix build -f channel:nixos-unstable \
      --arg crossSystem '{ config = "---"; }'

For instance you may want to cross-compile Firefox for ARM64 Linux. This is as easy as⁴:

$ nix build -f channel:nixos-unstable \
      --arg crossSystem '{ config = "arm64-unknown-linux-gnu"; }'

You can be much more specific with what you want through crossSystem. Many more combinations are possible, but they all revolve around that four-part string config listed. It corresponds to --- and is commonly called the LLVM triple⁵. The LLVM triple has become the standard way to specify systems across many free software toolchains including GCC, Binutils, Clang, libffi, etc. There is more information that can be specified in crossSystem & localSystem within Nixpkgs but this is not covered here as they are heavily dependent on the specific toolchain being used.

4. When things break

While the fundamentals of cross compiling in Nixpkgs are very good, individual packages will sometimes be broken. This is sometime because the package definition in Nixpkgs is incorrect. There are some common mistakes that occur that I want to cover here. First, the difference between ‘build-time’ vs ‘runtime’ dependencies⁶.

build-time dependencies: tools that will be run on the computer doing the cross compiling
runtime dependencies: libraries and tools that will run on the computer we are targeting.

In Nixpkgs, build-time dependencies should be put in nativeBuildInputs. Runtime dependencies should be put in buildInputs. Currently, this distinction has no effect on native compilation but it is crucial for correct cross-compilation. There are proposals to Nixpkgs to enforce the use of buildInputs as nativeBuildInputs even on native builds but this is yet to be agreed on⁷.

Sometimes your package will pull in a dependency indirectly so that dependency is not listed in buildInputs or nativeBuildInputs. This breaks the package splicing that goes on behind the scenes to make pick up the package set to get each package. To fix it, you will have to splice the package yourself. This is fairly straightforward. For examples, let’s say that your package depends on the pkgs.git git executable to be available through the GIT_CMD variable, which means it is not listed in nativeBuildInputs. In this case, you should instead refer to git as pkgs.buildPackages.git. This will pick up the build package set instead of the target package set.

There are a few more things that can go wrong within Nixpkgs. If you need to conditionally do something only when cross compiling (say a configure flag like --enable-cross-compilation), you should use stdenv.hostPlatform != stdenv.buildPlatform. If you want to check, for instance, that the platform you are building for is a Windows computer, just use stdenv.hostPlatform.isWindows, in the same way that you can also check for Linux with stdenv.hostPlatform.isLinux. These cases are often necessary, but remember they should only be used when absolutely needed. The more code we share between platforms, the more code is tested.

Sometimes packages are just not written in a cross-friendly way. This will usually happen just because the software author has not thought of how to handle cross compilation⁸. We want to work with software authors to make this process easier & contribute to the portability of free software. This takes time but we are definitely making progress. Contributions are always encouraged to the Nixpkgs repo.

5. Further reading

The concepts introduced here are also available in the Nixpkgs manual. These are the relevant sections/chapters:

GNU Automake also has a section on build vs. host vs. target. This will help clarify some of the naming conventions in Nixpkgs:

Automake Manual. Section 2.2.8, Cross-Compilation

Footnotes:

This is referred to as native compilation.

All examples are provided by the file lib/systems/examples.nix in Nixpkgs.

Cross-compilation to iPhone, unfortunately, requires that you download the unfree XCode environment. This is a consequence of Apple’s choices regarding what toolchains they allow.

⁴

In fact, each of these correspond to a value for crossSystem listed in lib/systems/examples.nix.

⁵

Of course there are 4 of them, so LLVM quadruple seems like a better name.

⁶

Like a few other parts of this article, this is somewhat of a simplification. There are many other types of dependencies but they all revolve around the build-time vs runtime distinction.

⁷

See strictDeps in pkgs/stdenv/generic/setup.sh.

⁸

Or even worse, they have thought about cross-compilation, but embraced many anti-patterns that break with Nixpkgs’ cross-compilation framework.

Channel Changing with Nix

Sun, 13 May 2018 00:00:00 -0500

1. Introduction to channels

One of the many underappreciated feature of Nix is its ability to travel back in time. Functional dependencies mean that you can easily pull in old releases of NixOS & Nixpkgs without changing your environment at all! It’s surprisingly easy in Nix 2.0 with its support for Import From Derivation.

First, I will provide some code to get us started. This Nix script is what I use as my “channel changer”. It bootstraps the use of old channels. In Nix-world, channels are just what we call the CI-tested branch of NixOS/Nixpkgs¹. The NixOS maintainers have been making releases consistently since 2013, so there is a lot of interesting history.

2. Channel changing

Here is my script that I will refer to later on in the post as “channels.nix” (be sure to try it out yourself!)²

let mapAttrs = f: set: builtins.listToAttrs (
      map (attr: { name = attr; value = f set.${attr}; })
          (builtins.attrNames set));
    channels = {
      aardvark    = "13.10";
      baboon      = "14.04";
      caterpillar = "14.12";
      dingo       = "15.09";
      emu         = "16.03";
      flounder    = "16.09";
      gorilla     = "17.03";
      hummingbird = "17.09";
      impala      = "18.03";
    };
in mapAttrs (v:
     import (builtins.fetchTarball
             "https://nixos.org/channels/nixos-${v}/nixexprs.tar.xz") {})
   channels

As you can see from the script there have been 9 releases in total. We use a different letter of the alphabet for each release, starting with A for Aardvark. We are now up to I for Impala³. New releases happen every 6 months with Aardvark released in December 2013. The releases are versioned as YY.MM which is a common practice for Linux distros.

3. ‘nix run’ magic

In my Nix script, I have created an “attribute” for each version that has been released. With Nix 2.0, it is very easy to run packages from them. Here is the command to run hello world from Hummingbird.

nix run -f channels.nix hummingbird.hello -c hello

Hello, world!

This has run the hello executable from the hummingbird release. Since you are most likely not running Hummingbird, it may take a while to the first time. However, once Nix has downloaded the needed files future execution will be instantaneous. The package is completely self-contained! To start, we will do examples in Impala (18.03) so that things go a little faster.

There are lots of packages in Nixpkgs so we don’t have to restrict ourselves to just hello. Let’s try out cowsay first.

nix run -f channels.nix impala.cowsay -c cowsay hello

 _______ 
< hello >
 ------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

There are many, many more of these commands. I’ve included a few below for you to try out on your own.

# Look up the weather
nix run -f channels.nix impala.curl -c curl wttr.in/seville

# Download music
nix run -f channels.nix impala.youtube-dl -c \
    youtube-dl -t --extract-audio \
    --audio-format mp3 \
    https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Go see a Star War
nix run -f channels.nix impala.telnet -c telnet towel.blinkenlights.nl 666
nix run -f channels.nix impala.sox -c bash -c \
    'for n in E2 A2 D3 G3 B3 E4;
     do play -n synth 4 pluck $n repeat 2;
     done'

# Play Nethack
nix run -f channels.nix impala.nethack -c nethack

# Get your fortune
nix run -f channels.nix impala.fortune -c fortune

4. The macOS+Nix odyssey

The fact that Nix works so well on macOS is a miracle in its own right. Apple has a proprietary ABI but Nix is intended to be used with free software. To get around this, many hacks are necessary including taking Apple’s standard C library⁴. Anyway, I was interested in how well the binaries produced by Nixpkgs hold up on my MacBook. For reference, here are the versions of macOS available when each release happened. Those familiar with macOS internals will remember some significant differences between these versions.

NixOS release	macOS release
Aardvark (13.10)	Mountain Lion (10.8)
Baboon (14.04)	Mavericks (10.9)
Caterpillar (14.12)	Yosemite (10.10)
Dingo (15.09)	Yosemite (10.10)
Emu (16.03)	El Capitan (10.11)
Flounder (16.09)	El Capitan (10.11)
Gorilla (17.03)	Sierra (10.12)
Hummingbird (17.09)	High Sierra (10.13)
Impala (18.03)	High Sierra (10.13)

So, my MacBook is running the latest macOS 10.13. Naturally we can test that Impala & Hummingbird will work correctly. hello is a good tester, of course, not comprehensive.

nix run -f channels.nix impala.hello -c hello

Hello, world!

nix run -f channels.nix hummingbird.hello -c hello

Hello, world!

But now let’s test Gorilla. It was released when macOS Sierra was still around but the ABI should be compatible.

nix run -f channels.nix gorilla.hello -c hello

dyld: Library not loaded: /usr/lib/system/libsystem_coretls.dylib
 Referenced from: /nix/store/v7i520r9c2p8z6vk26n53hfrxgqn8cl9-Libsystem-osx-10.11.6/lib/libSystem.B.dylib
 Reason: image not found
sh: line 1: 23628 Abort trap: 6           nix run -f channels.nix gorilla.hello -c hello

Oh no!

We can see that libSystem 10.11 has been downloaded for us⁵. However, libSystem is referring to an image that isn’t on our machine. libsystem_coretls.dylib must have existed in 10.11 macOS but been removed since then⁶.

At this point, it may look like Nixpkgs will be broken going backwards. But, I want to try Flounder just to see what happens.

nix run -f channels.nix flounder.hello -c hello

Hello, world!

Amazingly, it worked! I am still not sure what the differences are, but it seems that the older executable is still available. Let’s try out Emu to see what happens there.

nix run -f channels.nix emu.hello -c hello

builder for '/nix/store/s41jnb4kmxxbwj40c5l88k9ma0mwfy0b-hello-2.10.drv' failed due to signal 4 (Illegal instruction: 4)
error: build of '/nix/store/s41jnb4kmxxbwj40c5l88k9ma0mwfy0b-hello-2.10.drv' failed

Wow! Again we hit an issue. This is the infamouse Illegal instruction: 4 bug that is frequently hit in Nixpkgs⁷. It occurs when an executable uses instructions that have been blocked by the XNU kernel. This is usually because they are considered insecure so a patch is needed to fix it. We no longer support Emu, so this is probably the end of the line. Let’s try Dingo out just to be sure though.

nix run -f channels.nix dingo.hello -c hello

builder for '/nix/store/1cyagihl211vsis9bz09cqaz3h2yyc23-libxml2-2.9.3.drv' failed with exit code 77; last 10 log lines:
 checking for awk... awk
 checking whether make sets $(MAKE)... yes
 checking whether make supports nested variables... yes
 checking whether make supports nested variables... (cached) yes
 checking for gcc... gcc
 checking whether the C compiler works... no
 configure: error: in `/private/tmp/nix-build-libxml2-2.9.3.drv-0/libxml2-2.9.3':
 configure: error: C compiler cannot create executables
 See `config.log' for more details
 
cannot build derivation '/nix/store/jd4y5aps1z61jqbhsz1gy408zwwa49w4-clang-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n4q29z97dc1p9mqrn2ydhlfmsqwbgx0j-libarchive-3.1.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vh2bh7gaw2m0rgxscf3mhm1d3rz3xwfg-clang-wrapper-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zg90kfmf99h03z0fl03gw3gh105mb02c-cmake-3.3.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/45ndaky3079nd78042384f8hbidq7f7q-libc++abi-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/mmyz6rrddfahwl23i9d9vjh7wa8irp5k-stdenv-darwin-boot-3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/lqjabx84kndk75y8m0lq7zh5190k6zzz-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/lqjabx84kndk75y8m0lq7zh5190k6zzz-hello-2.10.drv' failed

This is a curious error because it is very different from the previous one. Back here we were still using Clang 3.3 & it looks like bootstrapping is failing on our newer machines. I was not using Nix at this time (late 2015), so I will have to defer to someone who remembers that time better. Let’s keep going.

nix run -f channels.nix caterpillar.hello -c hello

error: attribute 'hello' in selection path 'caterpillar.hello' not found

nix run -f channels.nix baboon.hello -c hello

error: attribute 'hello' in selection path 'baboon.hello' not found

nix run -f channels.nix aardvark.hello -c hello

error: attribute 'hello' in selection path 'aardvark.hello' not found

I’ve grouped them together because they have the same output. It appears that hello was not available back then! I’m not sure what is going on. Again, I will defer to someone else to explain why this happens. But, I know for a fact that GNU Hello is one of the first packages to be packaged in the Nix language⁸.

5. Conclusion

I wanted to also look at what happens on Linux when you go back through channels. I don’t have time currently so I am just including what I have. Anyway, if you are able to report back what happens on Linux when running these old channels, it would certainly be interesting.

My main goal was to just share some useful things in Nix that I don’t think many people outside of the core Nix community know about. Documentation has gotten better recently but lots of times people like to just read blog posts like this. Hopefully you got a feel for what can be done in Nix.

Footnotes:

The difference between NixOS & Nixpkgs can sometimes cause confusion especially because they are hosted in the same repository. We usually refer to NixOS for the Linux-specific distro while Nixpkgs refers to the cross-platform set of packages. Here I am referring to them collectively.

Note that the channel changing script is not necessary. You can always refer to the Nixpkgs version directly with the -f argument. The script is just an easy way to introduce people to the concept.

The in-development version of NixOS/Nixpkgs will be a J for Jackrabbit.

⁴

Apple’s C standard library is called libSystem. Note that unlike Glibc & Musl it contains much, much more than what is needed to compile simple C programs.

⁵

Note that the same libSystem is used for all of Nixpkgs to peliminate having to do SDK detection. Eventually we will update this to 10.12 or 10.13 but we prefer to stay a couple releases behind.

⁶

This is not a complete explanation, but the best I can do for those not aware of the internals of Nixpkgs.

⁷

See GitHub issue #17372.

⁸

See release 0.5.

2018 Summer

Sat, 31 Mar 2018 00:00:00 -0500

Quick post to annouce summer plans. I have accepted a summer internship position at Obsidian Systems in the greatest city in the world. Very excited to be working with a team of really smart people.

66% of Nixpkgs are up-to-date

Thu, 29 Mar 2018 00:00:00 -0500

Just wanted to congratulate everyone for all of the work in keeping Nixpkgs up-to-date!

Repology has just updated our stats and we have shown significant improvement. 66% of nixpkgs-unstable packages are up-to-date, with a total of 6112 up-to-date packages¹. Nixpkgs is slowly growing to match package sets that come in many of the big Linux distros. We now have more up-to-date packages than rolling release distros Arch (5924) and Manjaro (5708)! We certainly have a ways to go to compete with heavyweights like FreeBSD Ports (15410), DragonFly BSD Ports (14268), and Debian (14915) but these are ancient projects that have had a huge headstart. We are definitely making progress and it’s visible in the Repology graphs².

For comparison, in February, nixpkgs-unstable had 60% of packages up-to-date, with a total of 5439 up-to-date packages³. This kind of progress over 1 month is tremendously helpful. Remember that most of these updates will be included in 18.09, not the soon to be release 18.03. This protects users from potentially breaking changes as well as giving maintainers plenty of time to verify that the updates have not had any negative effects.

Ryan Mulligan’s nix-update has made updating packages much easier and I think it will be a game changer for Nixpkgs in the long run. There are still some kinks in it but I think it is slowly getting better. It has certainly been a headache to handle the huge increase in pull requests, but we are slowly clearing through that glut of outdated software. If an update to nixpkgs-unstable breaks something for you, please be sure to open an issue.

Footnotes:

https://repology.org/statistics/newest

https://repology.org/repository/nix_unstable

https://web.archive.org/web/20180216091523/https://repology.org/statistics/newest

nix-buffer: nix-shell in Emacs

Sun, 11 Feb 2018 00:00:00 -0600

Shea Levy has a wonderful package out for users of Nix and Emacs. It’s called nix-buffer and it greatly improves working on Nix stuff in Emacs. To try it just get it from MELPA by running M-x package-installnix-buffer.

To use it in a project you need to create a new file in your project directory called dir-locals.nix. It works a lot like .dir-locals.el but it allows you to evaluate Nix expressions.

To setup, just create dir-locals.nix that looks like this:

let pkgs = import  {};
in with pkgs; nixBufferBuilders.withPackages [ … ]

You can put anything from Nixpkgs inside of withPackages. That path is added to your exec-path in Emacs. Using this you can wrap your project in a container and avoid conflicts between project configurations!

There’s lot of future extensions for this. Ideally we could skip and the dir-locals.nix configuration and automatically detect what dependencies you need based on your default.nix file.

Why Nix?

Tue, 21 Nov 2017 00:00:00 -0600

THIS IS A WORK IN PROGRESS ESSAY

To someone new to it, Nix may not stand out as a package manager. We’re all used to how the average package manager works. You expect it to be able to install and uninstall apps and services and then mostly get out of the way. With that in mind, you look for ease-of-use, community support, and number of packages in your package manager. These are all important in any package manager and Nix compares well in some of these metrics¹. But just looking at this, you’ll miss what makes Nix useful.

Nix isn’t just a package manager. It’s a functional programming language that just happens to also be a really good package manager. At first, it might not make sense to have a programming language just for your package manager. The Nix model of package management takes some getting used to.

Packages are represented in Nix as a special data type called a derivation. The name derivation comes from the idea that each package is derived from some inputs like source code, libraries, and build tools². Each derivation has its own directory, unique to every unique derivation written in code.

Together, these ideas make Nix really useful. To demonstrate its usefulness, I’ve outlined three points where Nix is much better than an ordinary package manager. There are many others³ but these are the ones that I think are most convincing.

1. TODO Building

Package managers like RPM or Homebrew think of packages as tables.

2. TODO Using packages without installing

We are so used to installing software to use it. The Nix model gives you

3. TODO Rollbacks

4. Comments

Thanks for reading! I’ve been trying to improve my writing skills. This blog post is part of that effort. Feedback on this is welcome. My email is mjbauer95@gmail.com.

Footnotes:

Not all, but we’re improving every day. Usability is still a major issue.

For more features of Nix, you can checkout the Nix website.

Reproducible résumés

Mon, 13 Nov 2017 00:00:00 -0600

Recently, ‘reproducible’ papers have appeared in Academia. The idea is to make all of a paper’s data and code openly available so that readers can reproduce its findings. For software-heavy papers, it makes perfect sense to include code in papers because the code is often as important as the paper’s text.

Outside of Academia, there’s not as much interest in this idea of reproducibility. That’s unfortunate because it’s a really neat idea.

To make this idea more accessible for those outside Academia, I’m going to introduce a new kind of reproducibility. Everyone in the working world needs to apply for a job at some point. So, let’s figure out how to make a résumé, reproducible. This document will be all that is needed to generate my résumé. With some minor adjustments you can use it to generate your own.

At the end of this guide, you should have all of the information needed produce a PDF version of your résume.

1. The Résumé

LaTeX makes it very easy to generate Résumés. It provides extremely powerful typesetting features and can output PDF. Some of the syntax is scary to the unexperienced but I’ve tried to keep it as simple as possible.

LaTeX syntax can be a little confuging at first but basically everything is made up of commands. Each command starts with a \ followed by a command name. Each command can take a variable number of arguments and they are specified with { and }. LaTeX is made up a series of packages that provide these commands and do most of the work of typography automatically.

1.1. Document class

Every LaTeX document has a document class. Think of it like the CSS to the Résumé’s HTML. Classes tells LaTeX how to layout the document and there are a few builtin ones like ‘article’ that work fairly well for basic document. I’ve built up my own class that works well for résumés. It’s largely based off of @posquit0’s Awesome-CV but with some minor adjustments to make it a little cleaner. In addition, I’ve followed some of Matthew Butterick’s guide on résumés. You can read about them at Practical Typography.

I’ve make it available at resume.cls of this repo’s root but I won’t include it in this blog post (but it’s still in the ./README.html file).

1.2. Header

Every LaTeX document has a document class. For this document, I will use a custom resume class.

\documentclass[11pt]{resume}

Next, I’ll provide some semantic information. Each of these commands is defined in the résumé class. You can make your own commands but I’ve tried to make this document as simple as possible.

All of the following elements correspond to my name, address, mobile, email and homepage. The contents are not parsed by LaTeX so you could put anything here, but you should aim to make each command as semantic as possible.

My personal info follows,

\name{Matthew}{Bauer}
\address{10118 GARNETT ST, OVERLAND PARK KS 66214}
\mobile{(913) 671 0636}
\email{mjbauer95@gmail.com}
\homepage{www.matthewbauer.us}

This begins the document. The command makecvheader is from resume.cls and will setup the basic template.

\begin{document}

\makecvheader[C]

Everything that follows is my own résume, but the ideas should be fairly self-explanatory.

1.3. Education

This LaTeX class defines \cvsection to separate sections of the résumé by a horizontal line and some blank space. The only argument for \cvsection sets the contents of the section header.

I start my résume with the ‘Eduction’ section.

\cvsection{Education}

\begin{cventries}

Each cvsection is composed of multiple cventries.

\cventry
  {University of Kansas}
  {Lawrence, KS, USA}
  {B.S. in Computer Science}
  {Aug. 2015 – Exp. Dec. 2018}
  {
    \begin{cvitems}
    \item { Participated in ACM and hackathon competitions }
    \item { Coursework includes Software Engineering, Programming Languages,
        and Communication Networks }
    \end{cvitems}
  }

cventry can be though of as a function that takes a variable 5 arguments. The first will be the heading used. The second will be on the right-hand side in italics. The third will be in italics immediately below the first. The fourth will be on the right-hand side of the third argument. The last argument provides items for the entry.

For my résumé, I have chosen to use each of these arguments for certain purposes. It’s not necessary to do the same, but you should use a consistent style throughout your résumé.

left	right
argument 1	argument 2
argument 3	argument 4

For me, they correspond to,

left	right
Company or University	Location
Position or degree	Start date - end date

\end{cventries}

The next section has work experience. It follows the same pattern outlined in the Education section.

1.4. Work Experience

\cvsection{Work Experience}

\begin{cventries}

\cventry
  {Amazon.com, Inc.}
  {Seattle, WA, USA}
  {SDE Intern}
  {Summer 2017}
  {
    \begin{cvitems}
    \item { Worked on Mobile Identity team which manages the login screens for
            Amazon apps }
    \item { Project made it easier for teams to register new devices through
            Identity Services }
    \item { Used Agile development principles in design and development of
            project }
    \end{cvitems}
  }

\cventry
  {Lexmark Enterprise Software}
  {Lenexa, KS, USA}
  {Software Engineer Intern}
  {Summer 2015, Summer 2016}
  {
    \begin{cvitems}
    \item { Worked on the Client Architecture team which builds the JavaScript
        web framework which other teams use to build enterprise solutions }
    \item { Participated in high level design decision conversations }
    \item { Project moved the web framework away from in-house solutions to
        better maintained open source projects while preserving legacy
        compatibility }
    \item { Asked to return after impressive first year }
    \end{cvitems}
  }

\cventry
  {Together+Clinic}
  {Lincoln, NE, USA}
  {Design Studio Intern}
  {Spring 2015}
  {
    \begin{cvitems}
    \item { Startup building web app to let doctors track patients recovering
        from surgery without frequent checkup visits }
    \item { The web interface is used by both patients to record progress and
        doctors to track progress }
    \item { Team used Scrum development principles for quick response and user
        focused design }
    \end{cvitems}
  }

Here we end cventries.

\end{cventries}

1.5. Honors & Awards

Again we must define a new section, this time for honors and awards.

\cvsection{Honors \& Awards}
\begin{cvhonors}

\cvhonor
{3rd Place}
{JayHacks Hackathon}
{Lawrence, KS, USA}
{2017}

\cvhonor
{Grand Prize}
{Google Code-in}
{Mountain View, CA, USA}
{2013}

\end{cvhonors}

1.6. Footer

The makecvfooter command gives a nice footer that will be put at the bottom of each page. This can give us the document title and page numbering. In addition, the LastPage command will tell us how many pages there are in case we misplace a page while printing.

\makecvfooter
{BAUER}
{\thepage}
{\pageref{LastPage}}

\end{document}

2. Building it

Nix makes it possible to make this Résumé truly reproducible. Nix is a purely functional package manager. This means that each package is defined in a functional language and we have much more powerful tools at our disposal.

Nix can be installed on both Linux and macOS machines. It is fairly easy to setup, provided you have sudo access. Run the following and follow some simple steps to get Nix working,

curl https://nixos.org/nix/install | sh

More information on Nix is available from the Nix homepage. On the next page, I’ll explain how build this résumé using Nix.

2.1. `resume.nix`

To start, we’ll this need to pull in Nixpkgs. Nixpkgs provides a set of packages for Nix to use. Because Nix is functional, we’ll make nixpkgs an optional argument if we ever want to work with multiple package set versions.

{nixpkgs ? }: with import nixpkgs {};

This syntax may be a little hard to understand for users new to Nix. {}: declares a function. This particular function will take up the entire file and Nix will autocall it when no arguments are necessary. This particular function has one arguments, nixpkgs, that refers to the package set being used. To make things easier we provide a default after the ? symbol. refers to the nixpkgs channels that the user has setup. It can be updated with,

nix-channel --update

Giving us a potentially newer version of Nixpkgs and its software to work with.

Almost everything in Nix is a derivation (including Nix itself). Each derivation has its own store path so we can reference it through

stdenv.mkDerivation {
  name = "resume";
  src = ./.;

We’ll name this derivation resume and tell it to use the files in the current directory as source.

buildInputs = [
  (texlive.combine {
    inherit (texlive) scheme-basic xetex xetex-def setspace fontspec
                      chktex enumitem xifthen ifmtarg filehook
                      upquote tools ms geometry graphics oberdiek
                      fancyhdr lastpage xcolor etoolbox unicode-math
                      ucharcat sourcesanspro tcolorbox pgf environ
                      trimspaces parskip hyperref url euenc
                      collection-fontsrecommended;
  })
];

Inputs in Nix are similar to dependencies in other package managers. Here, we list only one dependency which provides our LaTeX distribution. texlive.combine is a function that produces a derivation which will provide the xetex binary. Each attribute listed in between { and } will be passed as LaTeX packages to TeX Live. The inherit keyword tells Nix to pass everything after (texlive) as attributes of texlive to texlive.combine. Each one of those names listed should correspond to TeX Live packages that are needed to build the résumé PDF.

In the future, I’d like to get Tex Live to actually recognize the packages we are using within LaTeX, but nothing seems to exist to do this.

buildPhase = ''
  xelatex -file-line-error -interaction=nonstopmode resume.tex
'';

Here we actually build the xelatex file. These options make it easier to debug xelatex when something goes wrong and makes sure we don’t get xelatex doesn’t require any user input. It will produce a file called resume.pdf that we can use as a résumé.

installPhase = ''
  cp resume.pdf $out
'';

Finally, we copy this résumé to $out where the derivation will live.

2.2. Running the build

This entire document is built with ~org-mode~’s Babel engine. This means that we can generate the files needed to build the résumé from scratch. To do this, first we must clone this repository (if you haven’t already).

git clone https://github.com/matthewbauer/resume
cd resume

Next, we need to open this file in Emacs and generate the files (tangle it in Babel lingo). Run this now, if you haven’t already,

emacs README.org

Finally, let’s build these files. From Emacs, type the following: C-c C-v t (org-babel-tangle). This will take a little bit, but at the end of it you will have all of the files tangled inside README.org. You can build the résumé with,

nix-build resume.nix

3. Automating it

Sadly, Nix does not understand raw Org mode (yet). We need a bootstrap to generate a Nix script from this file to truly automate this. I’ve included it here for completeness, but you’ll need to generate it first before Nix will work. If you haven’t already, generate this in org-mode by moving the cursor into the src block below and pressing C-u C-c C-v t (org-babel-tangle). Alternatively, I’ve provided a pregenerated file at ./default.nix.

{nixpkgs ? }: with import nixpkgs {};
let

Again, we’re be defining a function. Now, we will be using the let…in syntax to define a derivation to use.

README = stdenv.mkDerivation {
  name = "README";
  unpackPhase = "true";
  buildInputs = [ emacs ];
  installPhase = ''
    mkdir -p $out
    cd $out
    cp -r ${./fonts} fonts
    cp ${./README.org} README.org
    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
    cp resume.nix default.nix
  '';
};

The README derivation builds all of the things contained within this README.org file. Almost every code block here will make a file that we’ll feed into Nix. Fonts are external to the README because they are binary and cannot be put in an Org file, but you can view them in ./fonts.

Now, we’ll create another derivation with Nix. This will utilize a little known feature of Nix called IFD. It might not make sense right now, but it runs the README derivation’s resume.nix file as its own Nix expression.

IFD stands for Import From Derivation. Basically, it means we can import data generated in one derivation, README, in Nix to generate another derivation. This will bootstrap the ./README.html and allow us to avoid generated files.

in import README {inherit nixpkgs;}

This can be thought of as a recursive call to Nix. It basically lets us use the Nix output of README.org as an input for Nix. The derivation will produce a ./result file that will contain the output of build.nix for the Building it section.

Now, we can finally build the Résumé! To do this, we just need to run nix-build from the command line.

nix-build

Look at ./result and it will be PDF file you can open.

3.1. Continous Integration

Travis makes it easy to run continuous integration on our résumé.

language: nix

Travis supports Nix projects out-of-the-box so all that’s really needed is the above. However, matrices are useful to make sure it runs on more than one machine and accross different versions.

script: nix-build --arg nixpkgs "builtins.fetchTarball \"$NIXPKGS\""

This line tells Travis what to build. The $NIXPKGS variable should become clear after reading the usage below.

We want to target Linux and macOS. This will make sure the build script is portable. Unfortunately, Nix does not support Windows systems and Travis does not support any BSDs.

os:
  - linux
  - osx

Each value of NIXPKGS corresponds to a tarball release of Nixpkgs. This means that we can avoid problems that arise when the Nixpkgs repo is broken. Any URL to a tarball with a Nixpkgs set will work, but in Nix we call these ‘channels’. Each channel has a set of tests that are required to run before a new release of the channel. You could actually point directly to Nixpkgs HEAD using GitHub’s archive URL but this would constantly break as you would frequently have to rebuild whole packages when you could just let Hydra do it for you. You can find a whole listing at NixOS.org but I have included the most recent channels below.

env:
  - NIXPKGS=nixos.org/channels/nixos-17.09/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixpkgs-17.09-darwin/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

Travis works with ‘matrices’ meaning that every attribute of one property (os) will get crossed with another property (env). We need to modify some of these to get a working matrix.

matrix:

We exclude some (os, env) pairs here. It doesn’t really make sense to use a Darwin channel on Linux or a NixOS channel on macOS.

exclude:
  - os: linux
    env: NIXPKGS=nixos.org/channels/nixpkgs-17.09-darwin/nixexprs.tar.xz
  - os: osx
    env: NIXPKGS=nixos.org/channels/nixos-17.09/nixexprs.tar.xz
  - os: osx
    env: NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz

We allow some failure for unstable branches. We don’t expect stable releases to always work.

allow_failures:
  - env: NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz
  - env: NIXPKGS=nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

This will end up building five résumés three on Linux machines and two on macOS machines. So, using ./.travis.yml, you can make Travis automatically build a resume.pdf every time you commit a change. You should be able to set this up yourself but alternatively you can look at my Travis dashboard.

4. Conclusion

More information on reproducible research is available at Reproducible Research. My hope is that eventually more things will become ‘reproducible’. Technologies like Nix and Babel make this fairly easy but they have not yet entered into the average Software Developer’s toolbelt. Reproducible projects may take longer to setup, but they lead to more robust software systems.

Résumé require a careful mix of informative content and flashy styling. Too much information and employers will be overwhelmed but too little and employers assume you are inexperienced. Likewise, . Perhaps eventually, software companies will read through résumés in org-mode instead of PDFs, but alas Silicon Valley has not yet reached this level of Nirvana.

I welcome everyone to fork the repo containing these files. You should be able to generate your own Résumé by modifying the contents of Semantic info and LaTeX document. Any contributions to the process of reproducible résumés are welcome and you can open them as issues under that GitHub repo. Alternatively, you can email me at mjbauer95@gmail.com.

bauer: an Emacs+Nix IDE

Mon, 06 Nov 2017 00:00:00 -0600

Note: the updated version of this is available at https://matthewbauer.us/bauer/

I’m publishing my IDE as a blog post. It’s all written in Org Babel.

This file generates an Emacs configuration. It can be considered an ‘Emacs+Nix’ IDE. That is, the Emacs configuration is integrated with hardcoded Nix store paths. This provides a kind of functional Emacs configuration.

Usage

Demo

If you already have Nix installed, you can demo this out really easily. Just run the following,

url="github.com/matthewbauer/bauer/archive/master.tar.gz" \
expr='import (builtins.fetchTarball "$url")' \
   nix-shell -p nix-bundle --run 'nix-run "$expr"'

Installing/upgrading

Run this from your shell:

curl https://matthewbauer.us/bootstrap.sh | sh

Alternatively, you can accomplish the same thing from Emacs by downloading installer.el, loading it (M‑x load‑file) and running ‘install’ (M‑x install). This process will take a little bit as everything is downloaded. After you restart Emacs, everything should be working.

Developing

To make changes to the IDE, it is recommended you setup your environment like so,

git clone https://github.com/matthewbauer/bauer ~/bauer
cd ~/bauer
nix-build
./result/run

The last line will spawn an Emacs frame in the Git repo. Make any changes you want to this document or any of the files in the list folder. Make sure you commit your changes afterward by typing C-c p v, then c-ac (using Magit, of course). If you have forked this repo on GitHub, you can add it by typing Mg then your GitHub username within Magit. To push to it, just type Pr then find your username in the list and press enter. Pull requests are welcome through GitHub!

w/o Nix usage

You can use bauer without Nix. This just gives you the unintegrated Emacs configuration. To get started, run the following.

mkdir -p ~/.emacs.d
git clone https://github.com/matthewbauer/bauer ~/.emacs.d/bauer

Then, add the following to your Emacs init file,

(defvar bauer-dir (expand-file-name ".nixpkgs" (getenv "HOME")))
(defvar bauer-org (expand-file-name "README.org" bauer-dir))
(setq package-enable-at-startup nil)
(autoload 'org-babel-tangle-file "ob-tangle")
(if (locate-library "default")
    (unless (featurep 'default) (load "default" t))
  (let ((default-directory bauer-dir))
    (add-to-list 'load-path (expand-file-name "lisp" bauer-dir))
    (org-babel-tangle-file bauer-org "README.el" "emacs-lisp")
    (load-file (expand-file-name "README.el" bauer-dir))))

Emacs Init file

This is the main file of the IDE. It will be loaded by Emacs on startup.

Verify Emacs version ≥ 25

Emacs 24 is unsupported currently. This will check to make sure Emacs 25+ is available.

(unless (>= emacs-major-version 25)
  (error "Need Emacs 25+ to work properly"))

EXPERIMENTAL Emacs 24 support

If Nix is unavailable and only Emacs 24 is install for you, then you can try the experimental v24 branch. From the Git root of this repository, just run:

git checkout v24

and make sure you have setup the Nix-less version in your init.el file (see w/o Nix usage directions).

Increase GC

(setq gc-cons-threshold most-positive-fixnum)
(add-hook 'after-init-hook
          (lambda ()
            (garbage-collect)
            (setq gc-cons-threshold
                  (car (get 'gc-cons-threshold 'standard-value)))))

Autoloads

(autoload 'tramp-tramp-file-p "tramp")
(eval-and-compile (autoload 'use-package-autoload-keymap "use-package"))
(autoload 'package-installed-p "package")

Custom config

set-defaults provides an easy way to override the default custom files. This means that when you customize a variable it will appear as ‘standard’ even though it’s not what the package originally defined as the default. This is useful for an Emacs distribution to provide better defaults while still letting the user override them. Look through the lispdoc of the package for documentation on how this works. Eventually, this will be added to MELPA for use in other Emacs distributions.

(require 'set-defaults)

Better defaults

These are some better defaults for Emacs. They shouldn’t require any packages to be installed to work (those go in use-package).

(set-defaults
 '(TeX-auto-save t)
 '(TeX-engine 'xetex)
 '(ad-redefinition-action 'accept)
 '(apropos-do-all t)
 '(async-shell-command-buffer 'new-buffer)
 '(auth-source-save-behavior t)
 '(auto-revert-check-vc-info t)
 '(auto-revert-verbose nil)
 '(auto-save-visited-file-name t)
 '(backward-delete-char-untabify-method 'hungry)
 '(backup-directory-alist `(("." .
                             ,(expand-file-name "backup"
                                                user-emacs-directory))))
 '(bookmark-save-flag t)
 '(c-syntactic-indentation nil)
 '(comint-process-echoes t)
 '(comint-input-ignoredups t)
 '(comint-prompt-read-only t)
 '(comint-scroll-show-maximum-output nil)
 '(company-auto-complete (lambda () (and (company-tooltip-visible-p)
                                         (company-explicit-action-p))))
 '(company-frontends '(company-pseudo-tooltip-unless-just-one-frontend
                       company-preview-frontend
                       company-echo-metadata-frontend))
 '(company-continue-commands
   '(not save-buffer
         save-some-buffers
         save-buffers-kill-terminal
         save-buffers-kill-emacs
         comint-previous-matching-input-from-input
         comint-next-matching-input-from-input))
 '(company-require-match nil)
 '(company-selection-wrap-around t)
 '(compilation-always-kill t)
 '(compilation-ask-about-save nil)
 '(compilation-auto-jump-to-first-error nil)
 '(compilation-environment '("TERM=xterm-256color"))
 '(compilation-scroll-output nil)
 '(compilation-skip-threshold 2)
 '(completions-format 'vertical)
 '(completion-cycle-threshold 5)
 '(counsel-find-file-at-point t)
 '(counsel-mode-override-describe-bindings t)
 '(create-lockfiles nil)
 '(cursor-in-non-selected-windows nil)
 '(custom-safe-themes t)
 '(custom-buffer-done-kill t)
 '(custom-file (expand-file-name "settings.el" user-emacs-directory))
 '(custom-search-field nil)
 '(create-lockfiles nil)
 '(checkdoc-spellcheck-documentation-flag t)
 '(delete-old-versions t)
 '(delete-by-moving-to-trash t)
 '(dired-auto-revert-buffer t)
 '(dired-hide-details-hide-symlink-targets nil)
 '(dired-dwim-target t)
 '(dired-listing-switches "-alhv")
 '(dired-omit-verbose nil)
 '(dired-omit-files "^\\.")
 '(dired-recursive-copies 'always)
 '(dired-recursive-deletes 'always)
 '(dired-subtree-line-prefix " ")
 '(dtrt-indent-verbosity 0)
 '(disabled-command-function nil)
 '(display-buffer-reuse-frames t)
 '(echo-keystrokes 0)
 '(enable-recursive-minibuffers t)
 '(erc-autoaway-idle-seconds 600)
 '(erc-autojoin-timing 'ident)
 '(erc-fill-prefix "          ")
 '(erc-insert-timestamp-function 'erc-insert-timestamp-left)
 '(erc-interpret-mirc-color t)
 '(erc-kill-buffer-on-part t)
 '(erc-kill-queries-on-quit t)
 '(erc-kill-server-buffer-on-quit t)
 '(erc-prompt (lambda nil (concat "[" (buffer-name) "]")))
 '(erc-prompt-for-password nil)
 '(erc-query-display 'buffer)
 '(erc-server-coding-system '(utf-8 . utf-8))
 '(erc-timestamp-format "%H:%M ")
 '(erc-timestamp-only-if-changed-flag nil)
 '(erc-try-new-nick-p nil)
 '(eshell-banner-message "")
 '(eshell-cd-on-directory t)
 '(eshell-cmpl-autolist t)
 '(eshell-cmpl-cycle-completions nil)
 '(eshell-cmpl-cycle-cutoff-length 2)
 '(eshell-cmpl-ignore-case t)
 '(eshell-cp-interactive-query t)
 '(eshell-cp-overwrite-files nil)
 '(eshell-default-target-is-dot t)
 '(eshell-destroy-buffer-when-process-dies t)
 '(eshell-highlight-prompt t)
 '(eshell-hist-ignoredups t)
 '(eshell-history-size 10000)
 '(eshell-list-files-after-cd t)
 '(eshell-ln-interactive-query t)
 '(eshell-mv-interactive-query t)
 '(eshell-output-filter-functions '(eshell-handle-ansi-color
                                    eshell-handle-control-codes
                                    eshell-watch-for-password-prompt
                                    eshell-truncate-buffer))
 '(eshell-plain-echo-behavior nil)
 '(eshell-review-quick-commands t)
 '(eshell-rm-interactive-query t)
 '(eshell-prompt-function
   (lambda () (concat
               (when (tramp-tramp-file-p default-directory)
                 (concat
                  (tramp-file-name-user
                   (tramp-dissect-file-name default-directory))
                  "@"
                  (tramp-file-name-real-host (tramp-dissect-file-name
                                              default-directory))
                  " "))
               (let ((dir (eshell/pwd)))
                 (if (string= dir (getenv "HOME")) "~"
                   (let ((dirname (file-name-nondirectory dir)))
                     (if (string= dirname "") "/" dirname))))
               (if (= (user-uid) 0) " # " " $ "))))
 '(eshell-visual-commands
   '("vi" "screen" "top" "less" "more" "lynx" "ncftp" "pine" "tin" "trn" "elm"
     "nano" "nethack" "telnet" "emacs" "emacsclient" "htop" "w3m" "links" "lynx"
     "elinks" "irrsi" "mutt" "finch" "newsbeuter" "pianobar"))
 '(eval-expression-print-length 20)
 '(eval-expression-print-level nil)
 '(explicit-shell-args '("-c" "export EMACS= INSIDE_EMACS=; stty echo; shell"))
 '(expand-region-contract-fast-key "j")
 '(fill-column 80)
 '(flycheck-check-syntax-automatically '(save
                                         idle-change
                                         mode-enabled
                                         new-line))
 '(flycheck-display-errors-function
   'flycheck-display-error-messages-unless-error-list)
 '(flycheck-idle-change-delay 0.001)
 '(flycheck-standard-error-navigation nil)
 '(flycheck-global-modes '(not erc-mode
                               message-mode
                               git-commit-mode
                               view-mode
                               outline-mode
                               text-mode
                               org-mode))
 '(flyspell-abbrev-p nil)
 '(flyspell-auto-correct nil)
 '(flyspell-highlight-properties nil)
 '(flyspell-incorrect-hook nil)
 '(flyspell-issue-welcome-flag nil)
 '(frame-title-format '(:eval
                        (if (buffer-file-name)
                            (abbreviate-file-name (buffer-file-name))
                          "%b")))
 '(global-auto-revert-non-file-buffers t)
 '(highlight-nonselected-windows nil)
 '(hideshowvis-ignore-same-line nil)
 '(history-delete-duplicates t)
 '(history-length 20000)
 '(hippie-expand-verbose nil)
 '(iedit-toggle-key-default nil)
 '(imenu-auto-rescan t)
 '(indicate-empty-lines t)
 '(indent-tabs-mode nil)
 '(inhibit-startup-screen t)
 '(inhibit-startup-echo-area-message t)
 '(initial-major-mode 'fundamental-mode)
 '(initial-scratch-message "")
 '(ispell-extra-args '("--sug-mode=ultra"))
 '(ispell-silently-savep t)
 '(ispell-quietly t)
 '(ivy-count-format "\"\"")
 '(ivy-display-style nil)
 '(ivy-minibuffer-faces nil)
 '(ivy-use-virtual-buffers t)
 '(ivy-fixed-height-minibuffer t)
 '(jit-lock-defer-time 0.01)
 '(js2-mode-show-parse-errors nil)
 '(js2-mode-show-strict-warnings nil)
 '(js2-strict-missing-semi-warning nil)
 '(kill-do-not-save-duplicates t)
 '(kill-whole-line t)
 '(load-prefer-newer t)
 '(mac-allow-anti-aliasing t)
 '(mac-command-key-is-meta t)
 '(mac-command-modifier 'meta)
 '(mac-option-key-is-meta nil)
 '(mac-option-modifier 'super)
 '(mac-right-option-modifier nil)
 '(mac-frame-tabbing t)
 '(mac-system-move-file-to-trash-use-finder t)
 '(magit-log-auto-more t)
 '(magit-clone-set-remote\.pushDefault t)
 '(magit-diff-options nil)
 '(magit-display-buffer-function 'magit-display-buffer-fullframe-status-v1)
 '(magit-ediff-dwim-show-on-hunks t)
 '(magit-fetch-arguments nil)
 '(magit-highlight-trailing-whitespace nil)
 '(magit-highlight-whitespace nil)
 '(magit-no-confirm t)
 '(magit-process-connection-type nil)
 '(magit-process-find-password-functions '(magit-process-password-auth-source))
 '(magit-process-popup-time 15)
 '(magit-push-always-verify nil)
 '(magit-save-repository-buffers 'dontask)
 '(magit-stage-all-confirm nil)
 '(magit-unstage-all-confirm nil)
 '(mmm-global-mode 'buffers-with-submode-classes)
 '(mmm-submode-decoration-level 2)
 '(minibuffer-prompt-properties '(read-only t
                                            cursor-intangible t
                                            face minibuffer-prompt))
 '(mwim-beginning-of-line-function 'beginning-of-line)
 '(mwim-end-of-line-function 'end-of-line)
 '(neo-theme 'arrow)
 '(neo-fixed-size nil)
 '(next-error-recenter t)
 '(notmuch-show-logo nil)
 '(nrepl-log-messages t)
 '(nsm-save-host-names t)
 '(ns-function-modifier 'hyper)
 '(ns-pop-up-frames nil)
 '(org-blank-before-new-entry '((heading) (plain-list-item)))
 '(org-export-in-background nil)
 '(org-log-done 'time)
 '(org-return-follows-link t)
 '(org-special-ctrl-a/e t)
 '(org-src-fontify-natively t)
 '(org-src-preserve-indentation t)
 '(org-src-tab-acts-natively t)
 '(org-support-shift-select t)
 '(parens-require-spaces t)
 '(package-archives '(("melpa-stable" . "http://stable.melpa.org/packages/")
                      ("melpa" . "https://melpa.org/packages/")
                      ("org" . "http://orgmode.org/elpa/")
                      ("gnu" . "https://elpa.gnu.org/packages/")
                      ))
 '(proof-splash-enable nil)
 '(projectile-globally-ignored-files '(".DS_Store" "TAGS"))
 '(projectile-enable-caching t)
 '(projectile-mode-line
   '(:eval (if (and (projectile-project-p)
                    (not (file-remote-p default-directory)))
               (format " Projectile[%s]" (projectile-project-name)) "")))
 '(projectile-ignored-project-function 'file-remote-p)
 '(projectile-switch-project-action 'projectile-dired)
 '(projectile-do-log nil)
 '(projectile-verbose nil)
 '(reb-re-syntax 'string)
 '(require-final-newline t)
 '(resize-mini-windows t)
 '(ring-bell-function 'ignore)
 '(rtags-completions-enabled t)
 '(rtags-imenu-syntax-highlighting 10)
 '(ruby-insert-encoding-magic-comment nil)
 '(sh-guess-basic-offset t)
 '(same-window-buffer-names
   '("*eshell*" "*shell*" "*mail*" "*inferior-lisp*" "*ielm*" "*scheme*"))
 '(save-abbrevs 'silently)
 '(save-interprogram-paste-before-kill t)
 '(savehist-additional-variables '(search-ring
                                   regexp-search-ring
                                   kill-ring
                                   comint-input-ring))
 '(savehist-autosave-interval 60)
 '(auto-window-vscroll nil)
 '(hscroll-margin 5)
 '(hscroll-step 5)
 '(scroll-preserve-screen-position 'always)
 '(send-mail-function 'smtpmail-send-it)
 '(sentence-end-double-space nil)
 '(set-mark-command-repeat-pop t)
 '(shell-completion-execonly nil)
 '(shell-input-autoexpand nil)
 '(sp-autoskip-closing-pair 'always)
 '(sp-hybrid-kill-entire-symbol nil)
 '(truncate-lines nil)
 '(tab-always-indent 'complete)
 '(term-input-autoexpand t)
 '(term-input-ignoredups t)
 '(term-input-ring-file-name t)
 '(tramp-default-proxies-alist '(((regexp-quote (system-name)) nil nil)
                                 (nil "\\`root\\'" "/ssh:%h:")
                                 (".*" "\\`root\\'" "/ssh:%h:")))
 '(tramp-default-user nil)
 '(text-quoting-style 'quote)
 '(tls-checktrust t)
 '(undo-limit 800000)
 '(uniquify-after-kill-buffer-p t)
 '(uniquify-buffer-name-style 'forward)
 '(uniquify-ignore-buffers-re "^\\*")
 '(uniquify-separator "/")
 '(use-dialog-box nil)
 '(use-file-dialog nil)
 '(use-package-always-defer t)
 '(use-package-enable-imenu-support t)
 '(use-package-expand-minimally nil)
 '(version-control t)
 '(vc-allow-async-revert t)
 '(vc-command-messages nil)
 '(vc-git-diff-switches '("-w" "-U3"))
 '(vc-follow-symlinks nil)
 '(vc-ignore-dir-regexp
   (concat "\\(\\(\\`"
           "\\(?:[\\/][\\/][^\\/]+[\\/]\\|/\\(?:net\\|afs\\|\\.\\.\\.\\)/\\)"
            "\\'\\)\\|\\(\\`/[^/|:][^/|]*:\\)\\)\\|\\(\\`/[^/|:][^/|]*:\\)"))
 '(view-read-only t)
 '(view-inhibit-help-message t)
 '(visible-bell nil)
 '(visible-cursor nil)
 '(woman-imenu t)
 '(whitespace-line-column 80)
 '(whitespace-auto-cleanup t)
 '(whitespace-rescan-timer-time nil)
 '(whitespace-silent t)
 '(whitespace-style '(face
                      trailing
                      lines
                      space-before-tab
                      empty
                      lines-style))
 )

Site paths

Now, pull in generated paths from site-paths.el. Nix will generate this file automatically for us and different Emacs variables will be set to their Nix store derivations. Everything should work fine if you don’t have this available, though. If you are in Emacs and already have the IDE install you can inspect this file by typing M-: (find-file (locate-library "site-paths")). It will look similar to a settings.el file where each line corresponds to a customizable variable. Unlike settings.el, each entry is path in the Nix store and we verify it exists before setting it.

(load "site-paths" :noerror)

Set environment

set-envs is provided by set-defaults. We can use it like custom-set-variables, just it calls setenv instead of setq. All of these entries correspond to environment variables that we want to always be set in the Emacs process.

(set-envs
 '("EDITOR" "emacsclient -nw")
 '("LANG" "en_US.UTF-8")
 '("LC_ALL" "en_US.UTF-8")
 '("NODE_NO_READLINE" "1")
 '("PAGER" "cat")
 )

Load custom file

This file allows users to override above defaults.

(load custom-file 'noerror)

Setup use-package

Now to get use-package we will require package.el and initialize it if site-paths is not setup (meaning we’re outside the Nix expression). Because site-paths should be available (unless you don’t have Nix), we can skip this step. All of this is marked ‘eval-and-compile’ to make sure the compiler picks it up on build phase.

So, there are basically two modes for using this configuration. One when packages are installed externally (through Nix) and another where they are installed internally. This is captured in the variable ‘needs-package-init’ which will be t when we want to use the builtin package.el and will be nli when we want to just assume everything is available.

(eval-and-compile
  (setq needs-package-init (and (not (locate-library "site-paths"))
                                (not (and (boundp 'use-package-list--is-running)
                                          use-package-list--is-running)))))

First handle using package.el.

(when needs-package-init
  (require 'package)
  (package-initialize)
  (unless (package-installed-p 'use-package)
    (package-refresh-contents)
    (package-install 'use-package)))

Actually require use-package,

(eval-and-compile
  (require 'use-package)
  (require 'bind-key))

Now let’s handle the case where all of the packages are already provided. Bascially, we’ll prevent use-package from running ‘ensure’ on anything.

(eval-and-compile
  (setq use-package-always-ensure needs-package-init)
  (when (not needs-package-init)
    (setq use-package-ensure-function 'ignore
          package-enable-at-startup nil)))

Key bindings

Using bind-key, setup some simple key bindings. None of these should overwrite Emacs’ default keybindings. Also, they should only require vanilla Emacs to work (non-vanilla Emacs key bindings should be put in their use-package declaration).

(bind-key "C-c C-u"         'rename-uniquely)
(bind-key "C-x ~"           (lambda () (interactive) (find-file "~")))
(bind-key "C-x /"           (lambda () (interactive) (find-file "/")))
(bind-key "C-c C-o"         'browse-url-at-point)
(bind-key "H-l"             'browse-url-at-point)
(bind-key "C-x 5 3"         'iconify-frame)
(bind-key "C-x 5 4"         'toggle-frame-fullscreen)
(bind-key "s-SPC"           'cycle-spacing)
(bind-key "C-c w w"         'whitespace-mode)

(bind-key ""      'other-window)
(bind-key "C-z"             'delete-other-windows)
(bind-key "M-g l"           'goto-line)
(bind-key "" 'backward-kill-sexp)
(bind-key "C-x t"           'toggle-truncate-lines)
(bind-key "C-x v H"         'vc-region-history)
(bind-key "C-c SPC"         'just-one-space)
(bind-key "C-c f"           'flush-lines)
(bind-key "C-c o"           'customize-option)
(bind-key "C-c O"           'customize-group)
(bind-key "C-c F"           'customize-face)
(bind-key "C-c q"           'fill-region)
(bind-key "C-c s"           'replace-string)
(bind-key "C-c u"           'rename-uniquely)
(bind-key "C-c z"           'clean-buffer-list)
(bind-key "C-c ="           'count-matches)
(bind-key "C-c ;"           'comment-or-uncomment-region)
(bind-key "C-c n"           'clean-up-buffer-or-region)
(bind-key "C-c d"           'duplicate-current-line-or-region)
(bind-key "M-+"             'text-scale-increase)
(bind-key "M-_"             'text-scale-decrease)

(bind-key "H-c"             'compile)
(bind-key "s-1"             'other-frame)
(bind-key ""      'toggle-frame-fullscreen)

(bind-key "s-C-"      'shrink-window-horizontally)
(bind-key "s-C-"     'enlarge-window-horizontally)
(bind-key "s-C-"      'shrink-window)
(bind-key "s-C-"        'enlarge-window)

(require 'iso-transl)
(bind-key "' /"       "′" iso-transl-ctl-x-8-map)
(bind-key "\" /"      "″" iso-transl-ctl-x-8-map)
(bind-key "\" ("      "“" iso-transl-ctl-x-8-map)
(bind-key "\" )"      "”" iso-transl-ctl-x-8-map)
(bind-key "' ("       "‘" iso-transl-ctl-x-8-map)
(bind-key "' )"       "’" iso-transl-ctl-x-8-map)
(bind-key "4 < -"     "←" iso-transl-ctl-x-8-map)
(bind-key "4 - >"     "→" iso-transl-ctl-x-8-map)
(bind-key "4 b"       "←" iso-transl-ctl-x-8-map)
(bind-key "4 f"       "→" iso-transl-ctl-x-8-map)
(bind-key "4 p"       "↑" iso-transl-ctl-x-8-map)
(bind-key "4 n"       "↓" iso-transl-ctl-x-8-map)
(bind-key ""    "⇓" iso-transl-ctl-x-8-map)
(bind-key ""  "↓" iso-transl-ctl-x-8-map)
(bind-key ""    "⇐" iso-transl-ctl-x-8-map)
(bind-key ""  "←" iso-transl-ctl-x-8-map)
(bind-key ""   "⇒" iso-transl-ctl-x-8-map)
(bind-key "" "→" iso-transl-ctl-x-8-map)
(bind-key ""      "⇑" iso-transl-ctl-x-8-map)
(bind-key ""    "↑" iso-transl-ctl-x-8-map)
(bind-key ","         "…" iso-transl-ctl-x-8-map)

Helpers

These utils are needed at init stage and should always appear before other use-package declarations.

(use-package add-hooks
  :commands (add-hooks add-hooks-pair))

(use-package hook-helpers
  :commands (create-hook-helper
             define-hook-helper
             hkhlp-normalize-hook-spec)
  :functions (make-hook-helper
              add-hook-helper
              hkhlp-update-helper))

Packages

Alphabetical listing of all Emacs packages needed by the IDE.

Rules: No packages on the top level should have the :demand keyword. Each package should be setup as either commands, hooks, modes, or key bindings. Defer timers are allowed but should be used sparingly. Currently, these packages need defer timers:

autorevert (1)
company (2)
delsel (2)
dtrt-indent (3)
flycheck (3)
savehist (4)
save-place (5)
which-key (3)
apropostriate (2)

To resort, go to one of the package group headings and type C-c ^ (the shortcut for org-sort).

Essentials

Some of these are included in Emacs, others aren’t. All of them are necessary for using Emacs as a full featured IDE.

ace window

(use-package ace-window
  :bind (("M-o" . other-window)
         ([remap next-multiframe-window] . ace-window)))

aggressive-indent

Automatically indent code as you type. Only enabled for Lisp currently.

(use-package aggressive-indent
  :commands aggressive-indent-mode
  :init (add-hooks '(((emacs-lisp-mode
                       inferior-emacs-lisp-mode
                       ielm-mode
                       lisp-mode
                       inferior-lisp-mode
                       lisp-interaction-mode
                       slime-repl-mode) . aggressive-indent-mode))))

buffer-move

(use-package buffer-move
  :bind
  (("" . buf-move-up)
   ("" . buf-move-down)
   ("" . buf-move-left)
   ("" . buf-move-right)))

Company

(use-package company
  :demand
  :bind (:map company-active-map
              ("TAB" .
               company-select-next-if-tooltip-visible-or-complete-selection)
              ("" .
               company-select-next-if-tooltip-visible-or-complete-selection)
              ("S-TAB" . company-select-previous)
              ("" . company-select-previous)
              ("C-n" . company-select-next)
              ("C-p" . company-select-previous)
              )
  :commands (company-mode
             global-company-mode
             company-auto-begin
             company-complete-common-or-cycle)
  :config
  (setq company-backends
        '((company-css :with company-dabbrev)
          (company-nxml :with company-dabbrev)
          (company-elisp :with company-capf)
          (company-eshell-history :with company-capf company-files)
          (company-capf :with company-files company-keywords)
          (company-etags company-gtags company-clang company-cmake
                         :with company-dabbrev)
          (company-semantic :with company-dabbrev company-capf)
          (company-abbrev company-dabbrev company-keywords)
          ))
  (global-company-mode 1)
  (add-hook 'minibuffer-setup-hook 'company-mode)
  (add-hook 'minibuffer-setup-hook
            (lambda () (setq-local company-frontends
                                   '(company-preview-frontend))))
  (advice-add 'completion-at-point :override 'company-complete-common-or-cycle))

company-eshell-history

(use-package company-eshell-history
  :ensure nil
  :commands company-eshell-history
  )

company-statistics

(use-package company-statistics
  :commands company-statistics-mode
  :init (add-hook 'company-mode-hook 'company-statistics-mode))

compile

(use-package compile
  :ensure nil
  :bind (("C-c C-c" . compile)
         ("M-O" . show-compilation)
         :map compilation-mode-map
         ("o" . compile-goto-error))
  :preface
  (defun show-compilation ()
    (interactive)
    (let ((compile-buf
           (catch 'found
             (dolist (buf (buffer-list))
               (if (string-match "\\*compilation\\*" (buffer-name buf))
                   (throw 'found buf))))))
      (if compile-buf
          (switch-to-buffer-other-window compile-buf)
        (call-interactively 'compile))))

  :config
  (create-hook-helper compilation-ansi-color-process-output ()
    :hooks (compilation-filter-hook)
    (ansi-color-process-output nil)
    (set (make-local-variable 'comint-last-output-start)
         (point-marker))))

Counsel

(use-package counsel
  :commands (counsel-descbinds counsel-grep-or-swiper)
  :bind* (([remap execute-extended-command] . counsel-M-x)
          ([remap find-file] . counsel-find-file)
          ([remap describe-function] . counsel-describe-function)
          ([remap describe-variable] . counsel-describe-variable)
          ([remap info-lookup-symbol] . counsel-info-lookup-symbol)
          (" l" . counsel-find-library)
          ("C-c j" . counsel-git-grep)
          ("C-c k" . counsel-rg)
          ("C-x l" . counsel-locate)
          ("C-M-i" . counsel-imenu)
          ("M-y" . counsel-yank-pop)
          ("C-c i 8" . counsel-unicode-char)
          )
  :init
  (bind-key* [remap isearch-forward] 'counsel-grep-or-swiper
             (executable-find "grep"))
  )

diff-hl

(use-package diff-hl
  :commands (diff-hl-dir-mode diff-hl-mode diff-hl-magit-post-refresh
                              diff-hl-diff-goto-hunk)
  :bind (:map diff-hl-mode-map
              (" " . diff-hl-diff-goto-hunk))
  :init
  (add-hook 'prog-mode-hook 'diff-hl-mode)
  (add-hook 'vc-dir-mode-hook 'diff-hl-mode)
  (add-hook 'dired-mode-hook 'diff-hl-dir-mode)
  (add-hook 'magit-post-refresh-hook 'diff-hl-magit-post-refresh)
  )

dired

(use-package dired
  :ensure nil
  :init (require 'dired)
  :bind (("C-c J" . dired-double-jump)
         :map dired-mode-map
         ("C-c C-c" . compile)
         ("r" . browse-url-of-dired-file)))

dired-column

(use-package dired-column
  :ensure nil
  :after dired
  :bind (:map dired-mode-map
              ("o" . dired-column-find-file)))

dired-imenu

(use-package dired-imenu
  :after dired)

dired-subtree

(use-package dired-subtree
  :after dired
  :bind (:map dired-mode-map
              ("" . dired-subtree-toggle)
              ("" . dired-subtree-cycle)))

dired-x

(use-package dired-x
  :ensure nil
  :after dired
  :commands (dired-omit-mode dired-hide-details-mode)
  :init
  (add-hook 'dired-mode-hook 'dired-omit-mode)
  (add-hook 'dired-mode-hook 'dired-hide-details-mode)
  :bind (("s-\\" . dired-jump-other-window)
         :map dired-mode-map
         (")" . dired-omit-mode)))

dtrt-indent

(use-package dtrt-indent
  :commands dtrt-indent-mode
  :demand
  :config (dtrt-indent-mode 1))

eldoc

Provides some info for the thing at the point.

(use-package eldoc
  :ensure nil
  :commands eldoc-mode
  :init
  (add-hooks '(((emacs-lisp-mode
                 eval-expression-minibuffer-setup
                 lisp-mode-interactive-mode
                 typescript-mode) . eldoc-mode))))

Emacs shell

(use-package eshell
  :ensure nil
  :bind (("C-c M-t" . eshell)
         ("C-c x" . eshell))
  :commands (eshell eshell-command eshell-bol)
  :init
  (use-package em-rebind
    :preface
    (defun eshell-eol ()
      "Goes to the end of line."
      (interactive)
      (end-of-line))
    :ensure nil
    :demand
    :config
    (setq eshell-rebind-keys-alist
          '(([(control 97)] . eshell-bol)
            ([home] . eshell-bol)
            ([(control 100)] . eshell-delchar-or-maybe-eof)
            ([backspace] . eshell-delete-backward-char)
            ([delete] . eshell-delete-backward-char)
            ([(control 119)] . backward-kill-word)
            ([(control 117)] . eshell-kill-input)
            ([tab] . completion-at-point)
            ([(control 101)] . eshell-eol))))
  (setq eshell-modules-list
        '(eshell-alias
          eshell-banner
          eshell-basic
          eshell-cmpl
          eshell-dirs
          eshell-glob
          eshell-hist
          eshell-ls
          eshell-pred
          eshell-prompt
          eshell-rebind
          eshell-script
          eshell-smart
          eshell-term
          eshell-tramp
          eshell-unix
          eshell-xtra)))

esh-help

(use-package esh-help
  :commands esh-help-eldoc-command
  :init (create-hook-helper esh-help-setup ()
          :hooks (eshell-mode-hook)
          (make-local-variable 'eldoc-documentation-function)
          (setq eldoc-documentation-function 'esh-help-eldoc-command)
          (eldoc-mode)))

em-dired

(use-package em-dired
  :ensure nil
  :commands (em-dired-mode em-dired-new)
  :bind (:map dired-mode-map
              ("e" . em-dired))
  :init
  (add-hook 'eshell-mode-hook 'em-dired-mode)
  (advice-add 'eshell :before 'em-dired-new))

Emacs speaks statistics

(use-package ess-site
  :ensure ess
  :no-require
  :commands R)

esup

(use-package esup
  :commands esup
  :preface
  (defun init-profile ()
    (interactive)
    (esup (locate-library "default"))))

flycheck

(use-package flycheck
  :demand
  :commands global-flycheck-mode
  :config (global-flycheck-mode))

flycheck-irony

This is currently disabled.

(use-package flycheck-irony
  :commands flycheck-irony-setup
  :init (add-hook 'flycheck-mode-hook 'flycheck-irony-setup))

flyspell

(use-package flyspell
  :ensure nil
  :preface (require 'ispell)
  :when (executable-find ispell-program-name)
  :commands (flyspell-mode flyspell-prog-mode)
  :config
  (setq flyspell-use-meta-tab nil)
  :init
  (add-hook 'text-mode-hook 'flyspell-mode)
  (add-hook 'prog-mode-hook 'flyspell-prog-mode))

gnus

(use-package gnus
  :ensure nil
  :commands gnus
  :init
  (add-hook 'gnus-group-mode-hook 'gnus-topic-mode)
  (add-hook 'dired-mode-hook 'turn-on-gnus-dired-mode))

god-mode

(use-package god-mode
  :bind (("" . god-local-mode)))

gud

(use-package gud
  :ensure nil
  :commands gud-gdb
  )

help

(use-package help
  :ensure nil
  :bind (:map help-map
              ("C-v" . find-variable)
              ("C-k" . find-function-on-key)
              ("C-f" . find-function)
              ("C-l" . find-library)
              :map help-mode-map
              ("g" . revert-buffer-no-confirm))
  :preface
  (defun revert-buffer-no-confirm (&optional ignore-auto)
    "Revert current buffer without asking."
    (interactive (list (not current-prefix-arg)))
    (revert-buffer ignore-auto t nil)))

helpful

(use-package helpful
  :bind (("C-h f" . helpful-callable)
         ("C-h v" . helpful-variable)))

ivy

(use-package ivy
  :bind (("" . ivy-resume)
         ([remap list-buffers] . ivy-switch-buffer)
         :map ivy-minibuffer-map
         ("" . abort-recursive-edit))
  :commands ivy-mode
  :init
  (require 'ivy nil t)
  (defvar projectile-completion-system)
  (defvar magit-completing-read-function)
  (defvar dumb-jump-selector)
  (defvar rtags-display-result-backend)
  (defvar projector-completion-system)
  (setq projectile-completion-system 'ivy
        magit-completing-read-function 'ivy-completing-read
        dumb-jump-selector 'ivy
        rtags-display-result-backend 'ivy
        projector-completion-system 'ivy)
  :config (ivy-mode 1))

kill-or-bury-alive

(use-package kill-or-bury-alive
  :bind (([remap kill-buffer] . kill-or-bury-alive)))

magit

(use-package magit
  :preface
  (defun magit-dired-other-window ()
    (interactive)
    (dired-other-window (magit-toplevel)))

  (defun magit-remote-github (username &optional args)
    (interactive (list (magit-read-string-ns "User name")
                       (magit-remote-arguments)))
    (let* ((url (magit-get "remote.origin.url"))
           (match (string-match "^https?://github\.com/[^/]*/\\(.*\\)" url)))
      (unless match
        (error "Not a github remote"))
      (let ((repo (match-string 1 url)))
        (apply 'magit-remote-add username (format "https://github.com/%s/%s"
                                                  username repo) args))))

  :commands (magit-clone
             magit-toplevel
             magit-read-string-ns
             magit-remote-arguments
             magit-get
             magit-remote-add
             magit-define-popup-action)

  :bind (("C-x g" . magit-status)
         ("C-x G" . magit-dispatch-popup)
         :map magit-mode-map
         ("C-o" . magit-dired-other-window))
  :init
  (defvar magit-last-seen-setup-instructions "1.4.0")
  :config
  (create-hook-helper magit-github-hook ()
    :hooks (magit-mode-hook)
    (magit-define-popup-action 'magit-remote-popup
      ?g "Add remote from github user name" #'magit-remote-github)))

mb-depth

(use-package mb-depth
  :ensure nil
  :commands minibuffer-depth-indicate-mode
  :init (add-hook 'minibuffer-setup-hook 'minibuffer-depth-indicate-mode))

mmm-mode

(use-package mmm-mode
  :commands mmm-mode
  :config
  (use-package mmm-auto
    :ensure nil
    :demand))

multiple-cursors

(use-package multiple-cursors
  :bind
  (("" . mc/mark-next-like-this)
   ("" . mc/mark-previous-like-this)
   ("C->" . mc/mark-next-like-this)
   ("C-<" . mc/mark-previous-like-this)
   ("M-" . mc/add-cursor-on-click)
   ("C-c C-<"     . mc/mark-all-like-this)
   ("C-!"         . mc/mark-next-symbol-like-this)
   ("C-S-c C-S-c" . mc/edit-lines)))

mwim

(use-package mwim
  :bind (([remap move-beginning-of-line] . mwim-beginning-of-code-or-line)
         ([remap move-end-of-line] . mwim-end-of-code-or-line)))

org-mode

(use-package org
  :ensure org-plus-contrib
  :commands org-capture
  :bind* (("C-c c" . org-capture)
          ("C-c a" . org-agenda)
          ("C-c l" . org-store-link)
          ("C-c b" . org-iswitchb))
  :preface
  (defun org-completion-symbols ()
    (when (looking-back "=[a-zA-Z]+")
      (let (cands)
        (save-match-data
          (save-excursion
            (goto-char (point-min))
            (while (re-search-forward "=\\([a-zA-Z]+\\)=" nil t)
              (cl-pushnew
               (match-string-no-properties 0) cands :test 'equal))
            cands))
        (when cands
          (list (match-beginning 0) (match-end 0) cands)))))
(defun org-completion-refs ()
  (when (looking-back "\\\\\\(?:ref\\|label\\){\\([^\n{}]\\)*")
    (let (cands beg end)
      (save-excursion
        (goto-char (point-min))
        (while (re-search-forward "\\label{\\([^}]+\\)}" nil t)
          (push (match-string-no-properties 1) cands)))
      (save-excursion
        (up-list)
        (setq end (1- (point)))
        (backward-list)
        (setq beg (1+ (point))))
      (list beg end
            (delete (buffer-substring-no-properties beg end)
                    (nreverse cands))))))
  :init
  (add-hook 'org-mode-hook 'auto-fill-mode)
  (add-hook 'org-mode-hook
            (lambda ()
              (setq-local completion-at-point-functions
                          '(org-completion-symbols
                            org-completion-refs
                            pcomplete-completions-at-point))))
  :config
  (use-package ob-dot
    :ensure nil
    :demand)
  (use-package ox-rss
    :ensure nil
    :demand)
  (use-package ox-latex
    :ensure nil
    :demand)
  (use-package ox-beamer
    :ensure nil
    :demand)
  (use-package ox-md
    :ensure nil
    :demand)
  (use-package ox-reveal
    :disabled
    :demand)
  (use-package ox-pandoc
    :disabled
    :demand)
  (use-package ob-http
    :disabled
    :demand)
  (use-package org-brain
    :disabled
    :demand)
  (use-package org-projectile
    :disabled
    :demand)
  (use-package org-present
    :disabled
    :demand)
  (use-package org-ref
    :disabled
    :demand)
  (use-package org-autolist
    :disabled
    :demand)
  (use-package ox-tufte
    :disabled
    :demand)
  (use-package org-static-blog
    :demand)
  (org-babel-do-load-languages 'org-babel-load-languages
                               '((sh . t)
                                 (emacs-lisp . t)
                                 (dot . t)
                                 (latex . t)
                                 ))
  )

Projectile

Setup projectile and link it with some other packages. This also adds an easymenu to make the “Projectile” modeline clickable.

(use-package projectile
  :bind-keymap* (("C-c p" . projectile-command-map)
                 ("s-p" . projectile-command-map))
  :bind (:map projectile-command-map
         ("s r" . projectile-rg))
  :preface
  (defun projectile-rg ()
    "Run ripgrep in projectile."
    (interactive)
    (counsel-rg "" (projectile-project-root)))
  :commands (projectile-mode)
  :defer 1
  :config
  (put 'projectile-project-run-cmd 'safe-local-variable #'stringp)
  (put 'projectile-project-compilation-cmd 'safe-local-variable
       (lambda (a) (and (stringp a) (or (not (boundp 'compilation-read-command))
                                        compilation-read-command))))

  (projectile-mode)

  (use-package easymenu
    :ensure nil
    :config

    (easy-menu-define projectile-menu projectile-mode-map "Projectile"
      '("Projectile"
        :active nil
        ["Find file" projectile-find-file]
        ["Find file in known projects" projectile-find-file-in-known-projects]
        ["Find test file" projectile-find-test-file]
        ["Find directory" projectile-find-dir]
        ["Find file in directory" projectile-find-file-in-directory]
        ["Find other file" projectile-find-other-file]
        ["Switch to buffer" projectile-switch-to-buffer]
        ["Jump between implementation file and test file"
         projectile-toggle-between-implementation-and-test]
        ["Kill project buffers" projectile-kill-buffers]
        ["Recent files" projectile-recentf]
        ["Edit .dir-locals.el" projectile-edit-dir-locals]
        "--"
        ["Open project in dired" projectile-dired]
        ["Switch to project" projectile-switch-project]
        ["Switch to open project" projectile-switch-open-project]
        ["Discover projects in directory"
         projectile-discover-projects-in-directory]
        ["Search in project (grep)" projectile-grep]
        ["Search in project (ag)" projectile-ag]
        ["Replace in project" projectile-replace]
        ["Multi-occur in project" projectile-multi-occur]
        ["Browse dirty projects" projectile-browse-dirty-projects]
        "--"
        ["Run shell" projectile-run-shell]
        ["Run eshell" projectile-run-eshell]
        ["Run term" projectile-run-term]
        "--"
        ["Cache current file" projectile-cache-current-file]
        ["Invalidate cache" projectile-invalidate-cache]
        ["Regenerate [e|g]tags" projectile-regenerate-tags]
        "--"
        ["Compile project" projectile-compile-project]
        ["Test project" projectile-test-project]
        ["Run project" projectile-run-project]
        "--"
        ["Project info" projectile-project-info]
        ["About" projectile-version]
        ))))

Proof General

(use-package proof-site
  :ensure proofgeneral
  :no-require
  :disabled needs-package-init
  :commands (proofgeneral proof-mode proof-shell-mode))

Ripgrep
```
(use-package rg
  :commands rg)
```

Shell

(use-package shell
  :ensure nil
  :commands (shell shell-mode)
  :bind ("C-c C-s" . shell)
  :init
  (add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)
  (add-hook 'shell-mode-hook 'dirtrack-mode)
  (create-hook-helper use-histfile ()
    :hooks (shell-mode-hook)
    (turn-on-comint-history (getenv "HISTFILE"))))

smart-hungry-delete

(use-package smart-hungry-delete
  :commands (smart-hungry-delete-default-c-mode-common-hook
             smart-hungry-delete-default-prog-mode-hook
             smart-hungry-delete-default-text-mode-hook)
  :bind (:map prog-mode-map
              ("" . smart-hungry-delete-backward-char)
              ("C-d" . smart-hungry-delete-forward-char))
  :init
  (add-hook 'prog-mode-hook 'smart-hungry-delete-default-prog-mode-hook)
  (add-hook 'c-mode-common-hook 'smart-hungry-delete-default-c-mode-common-hook)
  (add-hook 'python-mode-hook 'smart-hungry-delete-default-c-mode-common-hook)
  (add-hook 'text-mode-hook 'smart-hungry-delete-default-text-mode-hook))

Smartparens

(use-package smartparens
  :commands (smartparens-mode
             show-smartparens-mode
             smartparens-strict-mode
             sp-local-tag
             sp-local-pair)
  :bind (:map smartparens-mode-map
              ("C-M-k" . sp-kill-sexp)
              ("C-M-f" . sp-forward-sexp)
              ("C-M-b" . sp-backward-sexp)
              ("C-M-n" . sp-up-sexp)
              ("C-M-d" . sp-down-sexp)
              ("C-M-u" . sp-backward-up-sexp)
              ("C-M-p" . sp-backward-down-sexp)
              ("C-M-w" . sp-copy-sexp)
              ("M-s" . sp-splice-sexp)
              ("C-}" . sp-forward-barf-sexp)
              ("C-{" . sp-backward-barf-sexp)
              ("M-S" . sp-split-sexp)
              ("M-J" . sp-join-sexp)
              ("C-M-t" . sp-transpose-sexp)
              ("C-M-" . sp-forward-sexp)
              ("C-M-" . sp-backward-sexp)
              ("M-F" . sp-forward-sexp)
              ("M-B" . sp-backward-sexp)
              ("C-M-a" . sp-backward-down-sexp)
              ("C-S-d" . sp-beginning-of-sexp)
              ("C-S-a" . sp-end-of-sexp)
              ("C-M-e" . sp-up-sexp)
              ("C-(" . sp-forward-barf-sexp)
              ("C-)" . sp-forward-slurp-sexp)
              ("M-(" . sp-forward-barf-sexp)
              ("M-)" . sp-forward-slurp-sexp)
              ("M-D" . sp-splice-sexp)
              ("C-" . sp-down-sexp)
              ("C-"   . sp-up-sexp)
              ("M-" . sp-splice-sexp-killing-forward)
              ("M-"   . sp-splice-sexp-killing-backward)
              ("C-" . sp-forward-slurp-sexp)
              ("M-" . sp-forward-barf-sexp)
              ("C-"  . sp-backward-slurp-sexp)
              ("M-"  . sp-backward-barf-sexp)
              ("C-k"   . sp-kill-hybrid-sexp)
              ("M-k"   . sp-backward-kill-sexp)
              ("M-" . backward-kill-word)
              ("C-" . sp-backward-kill-word)
              ([remap sp-backward-kill-word] . backward-kill-word)
              ("M-[" . sp-backward-unwrap-sexp)
              ("M-]" . sp-unwrap-sexp)
              ("C-x C-t" . sp-transpose-hybrid-sexp)
              :map smartparens-strict-mode-map
              ([remap c-electric-backspace] . sp-backward-delete-char)
              :map emacs-lisp-mode-map
              (";" . sp-comment))
  :init
  (add-hooks '(((emacs-lisp-mode
                 inferior-emacs-lisp-mode
                 ielm-mode
                 lisp-mode
                 inferior-lisp-mode
                 lisp-interaction-mode
                 slime-repl-mode
                 eval-expression-minibuffer-setup) . smartparens-strict-mode)))
  (add-hooks '(((emacs-lisp-mode
                 inferior-emacs-lisp-mode
                 ielm-mode
                 lisp-mode
                 inferior-lisp-mode
                 lisp-interaction-mode
                 slime-repl-mode) . show-smartparens-mode)))
  (add-hooks '(((web-mode
                 nxml-mode
                 html-mode) . smartparens-mode)))
  :config
  (use-package smartparens-html
    :ensure nil
    :demand)
  (use-package smartparens-config
    :ensure nil
    :demand)

  (sp-with-modes 'org-mode
    (sp-local-pair "*" "*"
                   :actions '(insert wrap)
                   :unless '(sp-point-after-word-p sp-point-at-bol-p)
                   :wrap "C-*" :skip-match 'sp--org-skip-asterisk)
    (sp-local-pair "_" "_" :unless '(sp-point-after-word-p) :wrap "C-_")
    (sp-local-pair "/" "/" :unless '(sp-point-after-word-p)
                   :post-handlers '(("[d1]" "SPC")))
    (sp-local-pair "~" "~" :unless '(sp-point-after-word-p)
                   :post-handlers '(("[d1]" "SPC")))
    (sp-local-pair "=" "=" :unless '(sp-point-after-word-p)
                   :post-handlers '(("[d1]" "SPC")))
    (sp-local-pair "«" "»"))

  (sp-with-modes
      '(java-mode c++-mode)
    (sp-local-pair "{" nil :post-handlers '(("||\n[i]" "RET")))
    (sp-local-pair "/*" "*/" :post-handlers '((" | " "SPC")
                                              ("* ||\n[i]" "RET"))))

  (sp-with-modes '(markdown-mode gfm-mode rst-mode)
    (sp-local-pair "*" "*" :bind "C-*")
    (sp-local-tag "2" "**" "**")
    (sp-local-tag "s" "```scheme" "```")
    (sp-local-tag "<"  "<_>" "" :transform 'sp-match-sgml-tags))

  (sp-local-pair 'emacs-lisp-mode "`" nil :when '(sp-in-string-p))
  (sp-local-pair 'clojure-mode "`" "`" :when '(sp-in-string-p))
  (sp-local-pair 'minibuffer-inactive-mode "'" nil :actions nil)
  (sp-local-pair 'org-mode "~" "~" :actions '(wrap))
  (sp-local-pair 'org-mode "/" "/" :actions '(wrap))
  (sp-local-pair 'org-mode "*" "*" :actions '(wrap)))

sudo-edit

(use-package sudo-edit
  :bind (("C-c C-r" . sudo-edit)))

swiper
```
(use-package swiper)
```

term

(use-package term
  :ensure nil
  :commands (term-mode term-char-mode term-set-escape-char)
  :init
  (add-hook 'term-mode-hook (lambda ()
                              (setq term-prompt-regexp "^[^#$%>\n]*[#$%>] *")
                              (setq-local transient-mark-mode nil)
                              (auto-fill-mode -1)))
  :preface
  (defun my-term ()
    (interactive)
    (set-buffer (make-term "my-term" "zsh"))
    (term-mode)
    (term-char-mode)
    (term-set-escape-char ?\C-x)
    (switch-to-buffer "*my-term*"))
  :bind ("C-c t" . my-term))

tramp

(use-package tramp
  :ensure nil
  :commands (tramp-tramp-file-p
             tramp-file-name-user
             tramp-file-name-real-host
             tramp-dissect-file-name))

transpose-frame

(use-package transpose-frame
  :bind ("H-t" . transpose-frame))

try
```
(use-package try
  :commands try)
```

which-func

(use-package which-func
  :commands which-function-mode
  :ensure nil
  :demand
  :config (which-function-mode))

which-key

(use-package which-key
  :commands which-key-mode
  :demand
  :config (which-key-mode))

whitespace-cleanup-mode

(use-package whitespace-cleanup-mode
  :commands whitespace-cleanup-mode
  :init (add-hook 'prog-mode-hook 'whitespace-cleanup-mode))

whitespace

(use-package whitespace
  :ensure nil
  :commands whitespace-mode
  :init (add-hook 'prog-mode-hook 'whitespace-mode))

yafolding

(use-package yafolding
  :commands yafolding-mode
  :init (add-hook 'prog-mode-hook 'yafolding-mode))

Built-ins

These are available automatically, so these use-package blocks just configure them.

align

(use-package align
  :bind (("C-c [" . align-regexp))
  :commands align
  :ensure nil)

ansi-color

Get color/ansi codes in compilation mode.

(use-package ansi-color
  :ensure nil
  :commands ansi-color-apply-on-region
  :init (create-hook-helper colorize-compilation-buffer ()
          :hooks (compilation-filter-hook)
          (let ((inhibit-read-only t))
            (ansi-color-apply-on-region (point-min) (point-max)))))

autorevert

(use-package autorevert
  :ensure nil
  :demand
  :commands auto-revert-mode
  :init
  (add-hook 'dired-mode-hook 'auto-revert-mode)
  :config
  (global-auto-revert-mode t))

bug-reference

(use-package bug-reference
  :ensure nil
  :commands bug-reference-prog-mode
  :init (add-hook 'prog-mode-hook 'bug-reference-prog-mode))

bug-reference-github

(use-package bug-reference-github
  :commands bug-reference-github-set-url-format
  :init (add-hook 'prog-mode-hook 'bug-reference-github-set-url-format))

comint

(use-package comint
  :ensure nil
  :bind
  (:map comint-mode-map
        ("C-r"       . comint-history-isearch-backward-regexp)
        ("s-k"       . comint-clear-buffer)
        ("M-TAB"     . comint-previous-matching-input-from-input)
        ("" . comint-next-matching-input-from-input))
  :commands (comint-next-prompt
             comint-write-input-ring
             comint-after-pmark-p
             comint-read-input-ring
             comint-send-input)
  :preface
  (defun turn-on-comint-history (history-file)
    (setq comint-input-ring-file-name history-file)
    (comint-read-input-ring 'silent))
  :config
  (add-hook 'kill-buffer-hook 'comint-write-input-ring)
  (create-hook-helper save-history ()
    :hooks (kill-emacs-hook)
    (dolist (buffer (buffer-list))
      (with-current-buffer buffer (comint-write-input-ring)))))

delsel

(use-package delsel
  :ensure nil
  :demand
  :config (delete-selection-mode t))

edebug
```
(use-package edebug
  :ensure nil)
```

electric

Setup these modes:

electric-quote
electric-indent
electric-layout

(use-package electric
  :ensure nil
  :commands (electric-quote-mode electric-indent-mode electric-layout-mode)
  :init
  (add-hook 'prog-mode-hook 'electric-quote-mode)
  (add-hook 'prog-mode-hook 'electric-indent-mode)
  (add-hook 'prog-mode-hook 'electric-layout-mode))

elec-pair

Setup electric-pair-mode for prog-modes. Also disable it when smartparens is setup.

(use-package elec-pair
  :ensure nil
  :commands electric-pair-mode
  :init
  (add-hook 'prog-mode-hook 'electric-pair-mode)
  (add-hook 'smartparens-mode-hook (lambda () (electric-pair-mode -1))))

etags

(use-package etags
  :ensure nil
  :commands (tags-completion-table))

executable

(use-package executable
  :ensure nil
  :commands executable-make-buffer-file-executable-if-script-p
  :init
  (add-hook 'after-save-hook
            'executable-make-buffer-file-executable-if-script-p))

ffap
```
(use-package ffap
  :ensure nil
  )
```

goto-addr

(use-package goto-addr
  :ensure nil
  :commands (goto-address-prog-mode goto-address-mode)
  :init
  (add-hook 'prog-mode-hook 'goto-address-prog-mode)
  (add-hook 'git-commit-mode-hook 'goto-address-mode))

grep

(use-package grep
  :ensure nil
  :bind (("M-s d" . find-grep-dired)
         ("M-s F" . find-grep)
         ("M-s G" . grep)))

hippie-exp

(use-package hippie-exp
  :ensure nil
  :bind* (("M-/". hippie-expand)))

ibuffer

(use-package ibuffer
  :ensure nil
  :bind ([remap switch-to-buffer] . ibuffer))

imenu

imenu-anywhere

(use-package imenu-anywhere
  :bind (("C-c i" . imenu-anywhere)
         ("s-i" . imenu-anywhere)))

imenu-list

(use-package imenu-list
  :commands imenu-list)

newcomment

(use-package newcomment
  :ensure nil
  :bind ("s-/" . comment-or-uncomment-region))

notmuch

(use-package notmuch
  :commands notmuch)

(use-package pp
  :ensure nil
  :commands pp-eval-last-sexp
  :bind (([remap eval-expression] . pp-eval-expression))
  :init
  (global-unset-key (kbd "C-x C-e"))
  (create-hook-helper always-eval-sexp ()
    :hooks (lisp-mode-hook emacs-lisp-mode-hook)
    (define-key (current-local-map) (kbd "C-x C-e") 'pp-eval-last-sexp)))

prog-mode

(use-package prog-mode
  :ensure nil
  :commands (prettify-symbols-mode global-prettify-symbols-mode)
  :init
  (add-hook 'prog-mode-hook 'prettify-symbols-mode)
  (create-hook-helper prettify-symbols-prog ()
    ""
    :hooks (prog-mode-hook)
    (push '("<=" . ?≤) prettify-symbols-alist)
    (push '(">=" . ?≥) prettify-symbols-alist))
  (create-hook-helper prettify-symbols-lisp ()
    ""
    :hooks (lisp-mode-hook)
    (push '("/=" . ?≠) prettify-symbols-alist)
    (push '("sqrt" . ?√) prettify-symbols-alist)
    (push '("not" . ?¬) prettify-symbols-alist)
    (push '("and" . ?∧) prettify-symbols-alist)
    (push '("or" . ?∨) prettify-symbols-alist))
  (create-hook-helper prettify-symbols-c ()
    ""
    :hooks (c-mode-hook)
    (push '("<=" . ?≤) prettify-symbols-alist)
    (push '(">=" . ?≥) prettify-symbols-alist)
    (push '("!=" . ?≠) prettify-symbols-alist)
    (push '("&&" . ?∧) prettify-symbols-alist)
    (push '("||" . ?∨) prettify-symbols-alist)
    (push '(">>" . ?») prettify-symbols-alist)
    (push '("<<" . ?«) prettify-symbols-alist))
  (create-hook-helper prettify-symbols-c++ ()
    ""
    :hooks (c++-mode-hook)
    (push '("<=" . ?≤) prettify-symbols-alist)
    (push '(">=" . ?≥) prettify-symbols-alist)
    (push '("!=" . ?≠) prettify-symbols-alist)
    (push '("&&" . ?∧) prettify-symbols-alist)
    (push '("||" . ?∨) prettify-symbols-alist)
    (push '(">>" . ?») prettify-symbols-alist)
    (push '("<<" . ?«) prettify-symbols-alist)
    (push '("->" . ?→) prettify-symbols-alist))
  (create-hook-helper prettify-symbols-js ()
    ""
    :hooks (js2-mode-hook js-mode-hook)
    (push '("function" . ?λ) prettify-symbols-alist)
    (push '("=>" . ?⇒) prettify-symbols-alist)))

savehist

(use-package savehist
  :ensure nil
  :demand
  :commands savehist-mode
  :config (savehist-mode 1))

simple

(use-package simple
  :ensure nil
  :demand
  :bind
  (("C-`" . list-processes)
   :map minibuffer-local-map
   (""  . abort-recursive-edit)
   ("M-TAB"     . previous-complete-history-element)
   ("" . next-complete-history-element))
  :commands visual-line-mode
  :init
  (add-hook 'text-mode-hook 'visual-line-mode)
  :config
  (column-number-mode))

subword

(use-package subword
  :ensure nil
  :commands subword-mode
  :init (add-hook 'java-mode-hook 'subword-mode))

time

(use-package time
  :demand
  :config
  (display-time-mode)
  )

tooltip

(use-package tooltip
  :ensure nil
  :demand
  :config
  (tooltip-mode -1))

view

(use-package view
  :ensure nil
  :bind (:map view-mode-map
              ("n" . next-line)
              ("p" . previous-line)
              ("j" . next-line)
              ("k" . previous-line)
              ("l" . forward-char)
              ("f" . forward-char)
              ("b" . backward-char)))

windmove

(use-package windmove
  :ensure nil
  :bind (("" . windmove-down)
         ("" . windmove-up)
         ))

Programming languages

Each use-package declaration corresponds to major modes in Emacs lingo. Each language will at least one of these major modes as well as associated packages (for completion, syntax checking, etc.)

C/C++

(use-package cc-mode
  :ensure nil
  :mode (("\\.h\\(h?\\|xx\\|pp\\)\\'" . c++-mode)
         ("\\.m\\'" . c-mode)
         ("\\.c\\'" . c-mode)
         ("\\.cpp\\'" . c++-mode)
         ("\\.c++\\'" . c++-mode)
         ("\\.mm\\'" . c++-mode))
  :config
  (use-package c-eldoc
    :commands c-turn-on-eldoc-mode
    :init (add-hook 'c-mode-common-hook 'c-turn-on-eldoc-mode)))

irony

(use-package irony
  :commands irony-mode
  :init (add-hooks '(((c++-mode c-mode objc-mode) . irony-mode))))

irony-eldoc

(use-package irony-eldoc
  :commands irony-eldoc
  :init (add-hook 'irony-mode-hook 'irony-eldoc))

CoffeeScript

(use-package coffee-mode
  :mode (("\\.coffee\\'" . coffee-mode)))

CSS

(use-package css-mode
  :ensure nil
  :mode "\\.css\\'"
  :commands css-mode
  :config
  (use-package css-eldoc
    :demand)
  )

CSV

(use-package csv-mode
  :mode "\\.csv\\'")

ELF

(use-package elf-mode
  :commands elf-mode
  :init (add-to-list 'magic-mode-alist (cons "ELF" 'elf-mode)))

(use-package go-mode
  :mode "\\.go\\'")

go-eldoc

(use-package go-eldoc
  :commands go-eldoc-setup
  :init (add-hook 'go-mode-hook 'go-eldoc-setup))

HAML

(use-package haml-mode
  :mode "\\.haml\\'")

Haskell

intero

(use-package intero
  :commands intero-mode
  :preface
  (defun intero-mode-unless-global-project ()
    "Run intero-mode iff we're in a project with a stack.yaml"
    (unless (string-match-p
             (regexp-quote ".stack/global-project")
             (shell-command-to-string
              "stack path --project-root --verbosity silent"))
      (intero-mode)))
  :init
  (add-hook 'haskell-mode-hook 'intero-mode-unless-global-project)
  )

ghc
```
(use-package ghc)
```

haskell-mode

(use-package haskell-mode
  :mode (("\\.hs\\'" . haskell-mode)
         ("\\.cabal\\'" . haskell-cabal-mode))
  :commands haskell-indentation-moe
  :init
  (add-hook 'haskell-mode-hook 'haskell-indentation-mode)
  :config
  (use-package haskell-doc
    :ensure nil
    :demand))

Java

jdee

(use-package jdee
  :mode ("\\.java\\'" . jdee-mode)
  :commands jdee-mode
  :bind (:map jdee-mode-map
              ("" . jdee-open-class-at-event)))

JavaScript

indium

(use-package indium
  :mode ("\\.js\\'" . indium-mode)
  :commands (indium-mode indium-interaction-mode indium-scratch))

js2-mode

(use-package js2-mode
  :mode (("\\.js\\'" . js2-mode))
  :commands js2-imenu-extras-mode
  :init
  (add-hook 'js2-mode-hook 'js2-imenu-extras-mode))

js3-mode

(use-package js3-mode
  :commands js3-mode)

tern

(use-package tern
  :commands tern-mode
  :init (add-hook 'js2-mode-hook 'tern-mode))

JSON

(use-package json-mode
  :mode (("\\.bowerrc$"     . json-mode)
         ("\\.jshintrc$"    . json-mode)
         ("\\.json_schema$" . json-mode)
         ("\\.json\\'" . json-mode))
  :config
  (make-local-variable 'js-indent-level))

LaTeX

auctex

Auctex provides some helpful tools for working with LaTeX.

(use-package tex-site
  :ensure auctex
  :no-require
  :commands (TeX-latex-mode
             TeX-mode
             tex-mode
             LaTeX-mode
             latex-mode)
  :mode ("\\.tex\\'" . TeX-latex-mode))

Lisp

(use-package elisp-mode
  :ensure nil
  :interpreter (("emacs" . emacs-lisp-mode)))

cider
```
(use-package cider)
```
slime
```
(use-package slime)
```

ielm

(use-package ielm
  :ensure nil
  :bind ("C-c :" . ielm))

LLVM IR

This is currently disabled.

(use-package llvm-mode
  :mode "\\.ll\\'")

Lua

lua-mode

(use-package lua-mode
  :mode "\\.lua\\'")

Mach-O

(use-package macho-mode
  :commands macho-mode
  :ensure nil
  :init
  (add-to-list 'magic-mode-alist '("\xFE\xED\xFA\xCE" . macho-mode))
  (add-to-list 'magic-mode-alist '("\xFE\xED\xFA\xCF" . macho-mode))
  (add-to-list 'magic-mode-alist '("\xCE\xFA\xED\xFE" . macho-mode))
  (add-to-list 'magic-mode-alist '("\xCF\xFA\xED\xFE" . macho-mode)))

Makefile

make-mode

(use-package make-mode
  :ensure nil
  :init
  (add-hook 'makefile-mode-hook 'indent-tabs-mode))

Markdown

markdown-mode

(use-package markdown-mode
  :mode
  (("\\.md\\'" . gfm-mode)
   ("\\.markdown\\'" . gfm-mode))
  :config
  (bind-key "'" "’" markdown-mode-map
            (not (or (markdown-code-at-point-p)
                     (memq 'markdown-pre-face
                           (face-at-point nil 'mult))))))

Nix

(use-package nix-mode
  :mode "\\.nix\\'")

nix-buffer

(use-package nix-buffer
  :commands nix-buffer)

NROFF

(use-package nroff-mode
  :ensure nil
  :commands nroff-mode)

PHP

(use-package php-mode
  :mode "\\.php\\'")

Python

Anaconda

(use-package anaconda-mode
  :commands (anaconda-mode anaconda-eldoc-mode)
  :init
  (add-hook 'python-mode-hook 'anaconda-mode)
  (add-hook 'python-mode-hook 'anaconda-eldoc-mode))

python-mode

(use-package python
  :ensure nil
  :mode ("\\.py\\'" . python-mode)
  :interpreter ("python" . python-mode))

elpy

(use-package elpy
  :mode ("\\.py\\'" . elpy-mode))

Ruby

(use-package ruby-mode
  :ensure nil
  :mode ("\\.rb\\'" . ruby-mode)
  :interpreter ("ruby" . ruby-mode))

Rust

(use-package rust-mode
  :mode "\\.rs\\'")

SASS

(use-package sass-mode
  :mode "\\.sass\\'")

Scala

(use-package scala-mode
  :interpreter ("scala" . scala-mode))

SCSS

(use-package scss-mode
  :mode "\\.scss\\'")

Shell

(use-package sh-script
  :ensure nil
  :preface
  (defun shell-command-at-point ()
    (interactive)
    (let ((start-point (save-excursion
                         (beginning-of-line)
                         (point))))
      (shell-command (buffer-substring start-point (point)))))
  :mode (("\\.*shellrc$" . sh-mode)
         ("\\.*shell_profile" . sh-mode)
         ("\\.zsh\\'" . sh-mode))
  :bind (:map sh-mode-map
              ("C-x C-e" . shell-command-at-point)))

texinfo

(use-package texinfo
  :mode ("\\.texi\\'" . texinfo-mode))

TypeScript

(use-package typescript-mode
  :mode "\\.ts\\'")

tide

(use-package tide
  :commands (tide-setup tide-hl-identifier-mode)
  :init
  (add-hook 'typescript-mode-hook 'tide-setup)
  (add-hook 'typescript-mode-hook 'tide-hl-identifier-mode))

Web

(use-package web-mode
  :mode (("\\.erb\\'" . web-mode)
         ("\\.mustache\\'" . web-mode)
         ("\\.html?\\'" . web-mode)
         ("\\.php\\'" . web-mode)
         ("\\.jsp\\'" . web-mode)))

XML

(use-package nxml-mode
  :ensure nil
  :commands nxml-mode
  :init
  (defalias 'xml-mode 'nxml-mode))

YAML

(use-package yaml-mode
  :mode "\\.ya?ml\\'")

Personal

These are all available in ./lisp. Eventually they should go into separate repositories.

dired-column
em-dired
installer
macho-mode

nethack

(use-package nethack
  :commands nethack
  :ensure nil)

nix-fontify
set-defaults
use-package-list

Other

These should correspond to minor modes or helper functions. Some of them are more helpful than others but none are essential.

Most of these are available in MELPA.

gnuplot
```
(use-package gnuplot)
```
logview
```
(use-package logview)
```

anything

(use-package anything
  :commands anything)

apropospriate-theme

This is the theme I use. This has to be defered for some reason.

(use-package apropospriate-theme
  :demand
  :config (load-theme 'apropospriate-dark t))

bm
```
(use-package bm)
```

bool-flip

(use-package bool-flip
  :bind ("C-c C-b" . bool-flip-do-flip))

browse-at-remote

(use-package browse-at-remote
  :commands browse-at-remote)

copy-as-format

(use-package copy-as-format
  :bind (("C-c w s" . copy-as-format-slack)
         ("C-c w g" . copy-as-format-github)))

crux

(use-package crux
  :bind (("C-c D" . crux-delete-file-and-buffer)
         ("C-c C-e" . crux-eval-and-replace)
         ([shift return] . crux-smart-open-line)))

elfeed

(use-package elfeed
  :commands elfeed)

expand-region

(use-package expand-region
  :bind (("C-=" . er/expand-region)))

firestarter

(use-package firestarter
  :bind ("C-c m s" . firestarter-mode))

focus

(use-package focus
  :bind ("C-c m f" . focus-mode))

hl-todo

(use-package hl-todo
  :commands hl-todo-mode
  :init (add-hook 'prog-mode-hook 'hl-todo-mode))

hookify

(use-package hookify
  :commands hookify)

htmlize
```
(use-package htmlize)
```
mediawiki
```
(use-package mediawiki)
```

minimap

(use-package minimap
  :commands minimap-mode)

multi-term

(use-package multi-term
  :bind (("C-. t" . multi-term-next)
         ("C-. T" . multi-term)))

page-break-lines

(use-package page-break-lines
  :commands page-break-lines-mode
  :init (add-hooks '(((doc-mode
                       emacs-lisp-mode) . page-break-lines-mode))))

pandoc-mode

(use-package pandoc-mode
  :commands (pandoc-mode pandoc-load-default-settings)
  :init
  (add-hook 'markdown-mode-hook 'pandoc-mode)
  (add-hook 'pandoc-mode-hook 'pandoc-load-default-settings))

rainbow-delimiters

(use-package rainbow-delimiters
  :commands rainbow-delimiters-mode
  :init (add-hooks '(((emacs-lisp-mode
                       inferior-emacs-lisp-mode
                       ielm-mode
                       lisp-mode
                       inferior-lisp-mode
                       lisp-interaction-mode
                       slime-repl-mode) . rainbow-delimiters-mode))))

rainbow-mode

(use-package rainbow-mode
  :commands rainbow-mode
  :init (add-hooks '(((emacs-lisp-mode
                       inferior-emacs-lisp-mode
                       ielm-mode
                       lisp-mode
                       inferior-lisp-mode
                       lisp-interaction-mode
                       slime-repl-mode) . rainbow-mode))))

restclient

(use-package restclient
  :mode (("\\.rest\\'" . restclient-mode)
         ("\\.restclient\\'" . restclient-mode)))

shrink-whitespace

(use-package shrink-whitespace
  :bind ("H-SPC" . shrink-whitespace))

smart-shift

(use-package smart-shift
  :bind (("C-c " . smart-shift-left)
         ("C-c " . smart-shift-right)
         ("C-c " . smart-shift-up)
         ("C-c " . smart-shift-down)))

string-inflection

(use-package string-inflection
  :bind (("C-c r r" . string-inflection-all-cycle)
         ("C-c r c" . string-inflection-camelcase)
         ("C-c r l" . string-inflection-lower-camelcase)
         ("C-c r u" . string-inflection-underscore)
         ("C-c r k" . string-inflection-kebab-case)
         ("C-c r J" . string-inflection-java-style-cycle)))

vkill

(use-package vkill
  :bind ("C-x L" . vkill))

xah-math-input

(use-package xah-math-input
  :commands xah-math-input-mode)

xterm-color

(use-package xterm-color
  :commands xterm-color-filter
  :init
  (add-hook 'comint-preoutput-filter-functions 'xterm-color-filter)
  (setq comint-output-filter-functions
        (remove 'ansi-color-process-output comint-output-filter-functions)))

Profiles

All of these files live outside of Emacs but are necessary for a usable developer environment. They are basic shell profile and some git configuration scripts as well.

`.profile`

To use this, you must create a short ~/.profile file. Here is an example,

bootstrap=$HOME/.nix-profile/etc/profile
[ -f $bootstrap ] && source $bootstrap

Here we setup .profile. First, setup exports.

export LANG=en_US.UTF-8 \
       LC_ALL=en_US.UTF-8 \
       INFOPATH=$PREFIX/share/info \
       MANPATH=$PREFIX/share/man \
       DICPATH=$PREFIX/share/hunspell \
       CLICOLOR=1 \
       GREP_OPTIONS='--color=auto' \
       GREP_COLOR='3;33' \
       LC_COLLATE=C \
       HISTFILE=$HOME/.history \
       HISTSIZE=16000 \
       HISTFILESIZE=16000 \
       HISTCONTROL=ignoreboth \
       SAVEHIST=15000 \
       SHELL_SESSION_HISTORY=1

Then setup aliases.

alias ls="TERM=ansi ls --color=always" \
      l="ls -lF" \
      ..="cd .." \
      ...="cd ../.." \
      ....="cd ../../.." \
      .....="cd ../../../.." \
      tree='tree -Csuh' \
      grep="grep --color=auto"

Configure INSIDE_EMACS.

if [ "$TERM" = dumb ] && ! [ -z "$INSIDE_EMACS" ]; then
    export TERM=dumb-emacs-ansi \
           COLORTERM=1
fi

Define update_terminal_cwd function.

update_terminal_cwd () {
    local SEARCH=' '
    local REPLACE='%20'
    local PWD_URL="file://$HOSTNAME${PWD//$SEARCH/$REPLACE}"
    printf '\e]7;%s\a' "$PWD_URL"
}

`.bashrc`

To use this, you must create a short ~/.bash_profile file. Here is an example,

bootstrap=$HOME/.nix-profile/etc/bashrc
[ -f $bootstrap ] && source $bootstrap

Source profile found above.

source @out@/etc/profile

When TERM=dumb, just do a simple prompt.

case "$TERM" in
    dumb)
        PS1="\W > "
        return
        ;;
esac

Setup some bash-specific features.

shopt -s cdspell \
         cdable_vars \
         checkhash \
         checkwinsize \
         cmdhist \
         dotglob \
         extglob \
         histappend \
         histreedit \
         histverify \
         nocaseglob \
         no_empty_cmd_completion \
         sourcepath

Turn on notify, noclobber, ignoreeof, emacs. These are bash-specific.

set -o notify \
    -o noclobber \
    -o ignoreeof \
    -o emacs

Setup prompt.

PS1='\e[0;34m\u@\h:\e[0;36m\w \e[0;33m$ \e[0m'

Bind keys when we’re interactive.

if [[ $- == *i* ]]; then
    bind '"\e/": dabbrev-expand'
    bind '"\ee": edit-and-execute-command'
fi

Run the update_terminal_cwd command when we’re in Apple_Terminal. This will give us the working directory in the title window.

if [ "$TERM_PROGRAM" = Apple_Terminal ] && [ -z "$INSIDE_EMACS" ]; then
    PROMPT_COMMAND="update_terminal_cwd;$PROMPT_COMMAND"
    update_terminal_cwd
fi

`.zshrc`

To use this, you must create a short ~/.zshrc file. Here is an example,

bootstrap=$HOME/.nix-profile/etc/zshrc
[ -f $bootstrap ] && source $bootstrap

Setup ZSH profile. First, we just source the global profile.

source @out@/etc/profile

Handle dumb options.

case "$TERM" in
    dumb)
        unsetopt zle \
                 prompt_cr \
                 prompt_subst
        if whence -w precmd >/dev/null; then
            unfunction precmd
        fi
        if whence -w preexec >/dev/null; then
            unfunction preexec
        fi
        PS1='$ '
        return
        ;;
esac

Load up site-functions in ZSH.

fpath+=(@out@/share/zsh/site-functions)

Setup ZSH auto suggestions.

. @zsh-autosuggestions@/share/zsh-autosuggestions/zsh-autosuggestions.zsh

Turn on colors.

autoload -U colors && colors

Turn on ZSH-specific options.

setopt always_to_end \
       append_history \
       auto_cd \
       auto_menu \
       auto_name_dirs \
       auto_pushd \
       cdablevarS \
       complete_in_word \
       correct \
       correctall \
       extended_glob \
       extended_history \
       hist_expire_dups_first \
       hist_find_no_dups \
       hist_ignore_dups \
       hist_ignore_space \
       hist_reduce_blanks \
       hist_verify \
       inc_append_history \
       interactive_comments \
       long_list_jobs \
       multios \
       no_beep \
       prompt_subst \
       pushd_ignore_dups \
       pushdminus \
       share_history \
       transient_rprompt

Setup completions.

ZSH_COMPDUMP="${HOME}/.zcompdump-${SHORT_HOST}-${ZSH_VERSION}"
autoload -U compaudit compinit && compinit -d "${ZSH_COMPDUMP}"
zmodload -i zsh/complist

Zstyle completions.

zstyle ':vcs_info:*' actionformats \
       '%F{5}(%f%s%F{5})%F{3}-%F{5}[%F{2}%b%F{3}|%F{1}%a%F{5}]%f '
zstyle ':vcs_info:*' formats '%F{5}(%f%s%F{5})%F{3}-%F{5}[%F{2}%b%F{5}]%f '
zstyle ':vcs_info:*' enable git

zstyle ':completion:*' matcher-list 'r:|=*' 'l:|=* r:|=*'
zstyle ':completion:*' matcher-list 'm:{a-zA-Z-_}={A-Za-z_-}' 'r:|=*' 'l:|=*'
zstyle ':completion:*' list-colors ''
zstyle ':completion:*:*:kill:*:processes' list-colors \
       '=(#b) #([0-9]#) ([0-9a-z-]#)*=01;34=0=01'
zstyle ':completion:*:*:*:*:processes' command \
       "ps -u $USER -o pid,user,comm -w -w"
zstyle ':completion:*:cd:*' tag-order local-directories directory-stack \
       path-directories
zstyle ':completion::complete:*' use-cache 1
zstyle ':completion::complete:*' cache-path ~/.zsh/cache/$HOST
zstyle ':completion:*' select-prompt \
       '%SScrolling active: current selection at %p%s'
zstyle ':completion:*::::' completer _expand _complete _ignored _approximate
zstyle ':completion:*' menu select=1 _complete _ignored _approximate
zstyle ':completion:*:*:-subscript-:*' tag-order indexes parameters
zstyle ':completion:*' verbose yes
zstyle ':completion:*:descriptions' format '%B%d%b'
zstyle ':completion:*:messages' format '%d'
zstyle ':completion:*:warnings' format 'No matches for: %d'
zstyle ':completion:*:corrections' format '%B%d (errors: %e)%b'
zstyle ':completion:*' group-name ''
zstyle ':completion:*:functions' ignored-patterns '_*'
zstyle ':completion:*:scp:*' tag-order files users \
       'hosts:-host hosts:-domain:domain hosts:-ipaddr"IP\ Address *'
zstyle ':completion:*:scp:*' group-order files all-files users hosts-domain \
       hosts-host hosts-ipaddr
zstyle ':completion:*:ssh:*' tag-order users \
       'hosts:-host hosts:-domain:domain hosts:-ipaddr"IP\ Address *'
zstyle ':completion:*:ssh:*' group-order hosts-domain hosts-host users \
       hosts-ipaddr
zstyle '*' single-ignored show

Turn on prompt with colors.

PROMPT='%F{blue}%n@%m:%F{cyan}%c%F{yellow} $ %F{reset}'

ZSH key bindings.

if [ "$TERM" = xterm-256color ]; then
    bindkey "^[[H" beginning-of-line
    bindkey "^[[F" end-of-line
    bindkey "^[[3~" delete-char
fi

Setup Apple Terminal so that CWD is shown.

if [ "$TERM_PROGRAM" = Apple_Terminal ] && [ -z "$INSIDE_EMACS" ]; then
    autoload add-zsh-hook
    add-zsh-hook chpwd update_terminal_cwd
    update_terminal_cwd
fi

`etc-profile.sh`

This just sources everything in the /etc/profile.d dir. PREFIX can be used to reference the Nix output dir.

export PREFIX=@out@

This will source everything in /etc/profile.d.

if [ -d @out@/etc/profile.d ]; then
  for i in @out@/etc/profile.d/*.sh; do
    if [ -r $i ]; then
      source $i
    fi
  done
fi

`.gitignore`

Some basic gitignore paths.

*~
\#*\#
*.DS_Store

`.gitconfig`

[core]
        editor = emacsclient
        excludesfile = @gitignore@

[commit]
        gpgSign = true

[gpg]
        program = "@gnupg@/bin/gpg"

[push]
        default = simple

[pull]
        rebase = true

[alias]
        amend = commit --amend

[help]
        autcorrect = 1

[color]
        ui = true

Bootstrapping

`site-paths.el.in`

This file provides site-specific paths. However, it must be substituted in Nix before we can actually run it in Emacs. To prevent Emacs from trying to run this, I’ve set the syntax to text.

(require 'set-defaults)
(require 'subr-x)

output-directory points to the nix-profile directory created by Nix. Ideally, this could point to a Nix store path, but the order of building means that we don’t know this until too late.

(defvar output-directory (expand-file-name ".nix-profile" (getenv "HOME")))

Setup exec-path.

(setq exec-path (append `(,(expand-file-name "bin" output-directory)
                           "/usr/sbin" "/usr/bin" "/sbin" "/bin")
                        exec-path))

Setup man-path.

(defvar man-path `("/usr/share/man"
                   "/usr/local/share/man"
                   ,(expand-file-name "share/man" output-directory)))

This will setup cacert-file var,

(defcustom cacert-file "/etc/ssl/certs/ca-bundle.crt"
  "Path for SSL certificates."
  :group 'environment)

Set env vars provided by Nix,

(set-envs
 ;; `("NIX_SSL_CERT_FILE" ,cacert-file)
 `("NIX_PATH" "nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs")
 `("PATH" ,(string-join exec-path ":"))
 `("MANPATH" ,(string-join man-path ":"))
 )

Set paths provided by Nix,

(set-paths
 '(company-cmake-executable "@cmake@/bin/cmake")
 '(doc-view-dvipdf-program "@ghostscript@/bin/dvipdf")
 '(cacert-file "@cacert@/etc/ssl/certs/ca-bundle.crt")
 '(doc-view-ps2pdf-program "@ghostscript@/bin/ps2pdf")
 '(dired-touch-program "@coreutils@/bin/touch")
 '(dired-chmod-program "@coreutils@/bin/chmod")
 '(dired-chown-program "@coreutils@/bin/chown")
 '(dired-free-space-program "@coreutils@/bin/df")
 '(diff-command "@diffutils@/bin/diff")
 '(find-program "@findutils@/bin/find")
 '(epg-gpg-program "@gpg@/bin/gpg")
 '(epg-gpgconf-program "@gpg@/bin/gpgconf")
 '(epg-gpgsm-program "@gpg@/bin/gpgsm")
 '(explicit-shell-file-name "@bash@/bin/bash")
 '(flycheck-sh-bash-executable "@bash@/bin/bash")
 '(flycheck-sh-zsh-executable "@zsh@/bin/zsh")
 '(flycheck-perl-executable "@perl@/bin/perl")
 '(flycheck-go-golint-executable "@golint@/bin/golint")
 '(flycheck-python-flake8-executable "@flake8@/bin/flake8")
 '(flycheck-asciidoc-executable "@asciidoc@/bin/asciidoc")
 '(flycheck-less-executable "@lessc@/bin/lessc")
 '(flycheck-haskell-stack-ghc-executable "@stack@/bin/stack")
 '(flycheck-c/c++-gcc-executable "@gcc@/bin/gcc")
 '(flycheck-javascript-eslint-executable "@eslint@/bin/eslint")
 '(flycheck-javascript-jshint-executable "@jshint@/bin/jshint")
 '(flycheck-go-build-executable "@go@/bin/go")
 '(flycheck-go-test-executable "@go@/bin/go")
 '(flycheck-lua-executable "@lua@/bin/luac")
 '(flycheck-xml-xmllint-executable "@libxml2@/bin/xmllint")
 '(flycheck-perl-perlcritic-executable "@perlcritic@/bin/perlcritic")
 '(flycheck-html-tidy-executable "@tidy@/bin/tidy")
 '(fortune-dir "@fortune@/share/games/fortunes")
 '(fortune-file "@fortune@/share/games/fortunes/food")
 '(grep-program "@gnugrep@/bin/grep")
 '(ispell-program-name "@aspell@/bin/aspell")
 '(irony-cmake-executable "@cmake@/bin/cmake")
 '(jka-compr-info-compress-program "@ncompress@/bin/compress")
 '(jka-compr-info-uncompress-program "@ncompress@/bin/uncompress")
 '(irony-server-install-prefix "@irony@")
 '(jka-compr-dd-program "@coreutils@/bin/dd")
;; '(jdee-server-dir "@jdeeserver@")
 '(magit-git-executable "@git@/bin/git")
 '(markdown-command "@markdown2@/bin/markdown2")
 '(manual-program "@man@/bin/man")
 '(man-awk-command "@gawk@/bin/awk")
 '(man-sed-command "@gnused@/bin/sed")
 '(man-untabify-command "@coreutils@/bin/pr")
 '(nethack-executable "@nethack@/bin/nethack")
 '(org-pandoc-command "@pandoc@/bin/pandoc")
 '(pandoc-binary "@pandoc@/bin/pandoc")
 '(remote-shell-program "@openssh@/bin/ssh")
 '(ripgrep-executable "@ripgrep@/bin/rg")
 '(rtags-path "@rtags@/bin")
 '(sql-ingres-program "@parallel@/bin/sql")
 '(sql-interbase-program "@unixODBC@/bin/isql")
 '(sql-mysql-program "@mariadb@/bin/mysql")
 '(sql-ms-program "@freetds@/bin/osql")
 '(sql-postgres-program "@freetds@/bin/osql")
 '(sql-sqlite-program "@sqliteInteractive@/bin/sqlite3")
 '(tramp-encoding-shell "@bash@/bin/sh")
 '(tex-shell "@bash@/bin/sh")
 '(xargs-program "@findutils@/bin/xargs")
 '(vc-git-program "@git@/bin/git")
 '(gnutls "@gnutls@/bin/gnutls-cli")
 '(pdf2dsc-command "@ghostscript@/bin/pdf2dsc")
 '(preview-gs-command "@texlive@/bin/rungs")
 '(TeX-command "@texlive@/bin/tex")
 '(LaTeX-command "@texlive@/bin/latex")
 '(luatex-command "@texlive@/bin/luatex")
 '(xetex-command "@texlive@/bin/xetex")
 '(xelatex-command "@texlive@/bin/xelatex")
 '(makeinfo-command "@texinfoInteractive@/bin/makeinfo")
 '(pdftex-command "@texlive@/bin/pdftex")
 '(context-command "@texlive@/bin/context")
 '(bibtex-command "@texlive@/bin/bibtex")
 '(dvipdfmx-command "@texlive@/bin/dvipdfmx")
 '(makeindex-command "@texlive@/bin/makeindex")
 '(chktex-command "@texlive@/bin/chktex")
 '(lacheck-command "@texlive@/bin/lacheck")
 '(dvipdfmx-command "@texlive@/bin/dvipdfmx")
 '(dvips-command "@texlive@/bin/dvips")
 '(dvipng-command "@texlive@/bin/dvipng")
 '(ps2pdf-command "@ghostscript@/bin/ps2pdf")
 '(locate-executable "@findutils@/bin/locate")
 '(ag-executable "@ag@/bin/ag")
 '(intero-stack-executable "@intero@/bin/intero-nix-shim")
 '(notmuch-command "@notmuch@/bin/notmuch")
 '(org-export-async-init-file "@orginit@")
 )

Set some defaults that depend on the path variables below,

(set-defaults
 '(imap-ssl-program `(,(concat gnutls " --tofu -p %p %s")))
 '(tls-program (concat gnutls " --tofu -p %p %h"))
 '(preview-pdf2dsc-command
   (concat pdf2dsc-command " %s.pdf %m/preview.dsc"))
 '(preview-dvips-command
   (concat dvips-command " -Pwww %d -o %m/preview.ps"))
 '(preview-fast-dvips-command
   (concat dvips-command " -Pwww %d -o %m/preview.ps"))
 '(preview-dvipng-command
   (concat dvipng-command
           " -picky -noghostscript %d -o \"%m/prev%%03d.png\""))
 '(TeX-engine-alist `((xetex "XeTeX" xetex-command xelatex-command
                             xetex-command)
                      (luatex "LuaTeX" luatex-command
                              ,(concat luatex-command " --jobname=%s")
                              luatex-command)))
 '(TeX-command-list
   `(("TeX"
      "%(PDF)%(tex) %(file-line-error) %(extraopts) %`%S%(PDFout)%(mode)%' %t"
      TeX-run-TeX nil
      (plain-tex-mode ams-tex-mode texinfo-mode)
      :help "Run plain TeX")
     ("LaTeX" "%`%l%(mode)%' %t" TeX-run-TeX nil
      (latex-mode doctex-mode)
      :help "Run LaTeX")
     ("Makeinfo" ,(concat makeinfo-command " %(extraopts) %t")
      TeX-run-compile nil
      (texinfo-mode)
      :help "Run Makeinfo with Info output")
     ("Makeinfo HTML" ,(concat makeinfo-command " %(extraopts) --html %t")
      TeX-run-compile nil
      (texinfo-mode)
      :help "Run Makeinfo with HTML output")
     ("AmSTeX"
      ,(concat pdftex-command " %(PDFout) %(extraopts) %`%S%(mode)%' %t")
      TeX-run-TeX nil
      (ams-tex-mode)
      :help "Run AMSTeX")
     ("ConTeXt"
      ,(concat context-command " --once --texutil %(extraopts) %(execopts)%t")
      TeX-run-TeX nil
      (context-mode)
      :help "Run ConTeXt once")
     ("ConTeXt Full" ,(concat context-command " %(extraopts) %(execopts)%t")
      TeX-run-TeX nil
      (context-mode)
      :help "Run ConTeXt until completion")
     ("BibTeX" ,(concat bibtex-command " %s")
      TeX-run-BibTeX nil t :help "Run BibTeX")
     ("Biber" "biber %s" TeX-run-Biber nil t :help "Run Biber")
     ("View" "%V" TeX-run-discard-or-function t t :help "Run Viewer")
     ("Print" "%p" TeX-run-command t t :help "Print the file")
     ("Queue" "%q" TeX-run-background nil t
      :help "View the printer queue" :visible TeX-queue-command)
     ("File" ,(concat dvips-command " %d -o %f ")
      TeX-run-dvips t t :help "Generate PostScript file")
     ("Dvips" ,(concat dvips-command " %d -o %f ")
      TeX-run-dvips nil t :help "Convert DVI file to PostScript")
     ("Dvipdfmx" ,(concat dvipdfmx-command " %d")
      TeX-run-dvipdfmx nil t :help "Convert DVI file to PDF with dvipdfmx")
     ("Ps2pdf" ,(concat ps2pdf-command " %f")
      TeX-run-ps2pdf nil t :help "Convert PostScript file to PDF")
     ("Index" ,(concat makeindex-command " %s")
      TeX-run-index nil t :help "Run makeindex to create index file")
     ("upMendex" "upmendex %s"
      TeX-run-index t t :help "Run mendex to create index file")
     ("Xindy" "xindy %s"
      TeX-run-command nil t :help "Run xindy to create index file")
     ("Check" ,(concat lacheck-command " %s") TeX-run-compile nil
      (latex-mode)
      :help "Check LaTeX file for correctness")
     ("ChkTeX" ,(concat chktex-command " -v6 %s") TeX-run-compile nil
      (latex-mode)
      :help "Check LaTeX file for common mistakes")
     ("Spell" "(TeX-ispell-document \"\")"
      TeX-run-function nil t :help "Spell-check the document")
     ("Clean" "TeX-clean"
      TeX-run-function nil t :help "Delete generated intermediate files")
     ("Clean All" "(TeX-clean t)" TeX-run-function nil t
      :help "Delete generated intermediate and output files")
     ("Other" "" TeX-run-command t t :help "Run an arbitrary command")))
 '(counsel-grep-base-command
   (concat ripgrep-executable
           " -i -M 120 --no-heading --line-number --color never '%s' %s"))
 '(counsel-rg-base-command
   (concat ripgrep-executable " -i --no-heading --line-number %s ."))
 '(counsel-ag-base-command (concat ag-executable " --nocolor --nogroup %s"))
 '(org-preview-latex-process-alist
   `((dvipng :programs ("latex" "dvipng")
             :description "dvi > png"
             :message ""
             :image-input-type "dvi"
             :image-output-type "png"
             :image-size-adjust (1.0 . 1.0)
             :latex-compiler
             (,(concat LaTeX-command
                       " -interaction nonstopmode -output-directory %o %f"))
             :image-converter
             (,(concat dvipng-command
                       " -fg %F -bg %B -D %D -T tight -o %O %f")))))
 '(Info-directory-list `(,(expand-file-name "share/info" output-directory)))
 '(tramp-remote-path `(tramp-own-remote-path
                       "/run/current-system/sw/bin"
                       tramp-default-remote-path
                       "/bin"
                       "/usr/bin"
                       "/sbin"
                       "/usr/sbin"
                       "/usr/local/bin"
                       "/usr/local/sbin"
                       "/opt/bin"
                       "/opt/sbin"
                       ,(expand-file-name "bin" output-directory)
                       ))
 '(woman-manpath man-path)
 )

Provide site-paths,

(provide 'site-paths)

`bootstrap.sh`

#!/bin/sh

Install Nix.

if ! command -v nix-env >/dev/null 2>&1; then
    nix_installer=$(mktemp)
    curl -s https://nixos.org/nix/install > $nix_installer
    sh $nix_installer
fi

Pull into Git repo.

if [ -d .git ] && command -v git >/dev/null 2>&1
then
    git pull origin master || true
fi

Clone if default.nix doesn’t exist.

if ! [ -f default.nix ] &&  && command -v git >/dev/null 2>&1; then
    repo_dir=$(mktemp -d)
    git clone https://github.com/matthewbauer/bauer $repo_dir
    cd $repo_dir
fi

Install current directory

nix-env -if .

`runemacs.sh`

Cross-platform script to execute app.

#!/usr/bin/env sh

case $(uname) in
    Darwin)
        open @out@/Applications/Emacs.app
    ;;
    *)
        @out@/bin/emacs
    ;;
esac

Building

`default.nix`: the tangler

{ nixpkgs-url ? "nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz"
, nixpkgs ? builtins.fetchTarball nixpkgs-url
, nixpkgs' ? import nixpkgs {}
}: with nixpkgs';

Now let’s tangle README.org…

import (runCommand "README" { buildInputs = [ emacs ]; } ''
mkdir -p $out
cd $out
cp -r ${./lisp} $out/lisp
cp ${./README.org} README.org
emacs --batch --quick \
      -l ob-tangle \
      --eval "(org-babel-tangle-file \"README.org\")"
cp bauer.nix default.nix
'') {inherit nixpkgs';}

`bauer.nix`: the build script

nixpkgs should just be a Nix path to the Nixpkgs package set. nixpkgs' will be nixpkgs instantiated.

{ nixpkgs-url ? "nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz"
, nixpkgs ? builtins.fetchTarball nixpkgs-url
, nixpkgs' ? import nixpkgs {}
}: with nixpkgs'; let

We start out by defining the Nix version to use. nixStable corresponds to the last released version, 1.11.

nix = nixStable;

Next we start defining some packages. R is one of the simpler ones right now, so let’s start with that.

rEnv = pkgs.rWrapper.override {
  packages = with pkgs.rPackages; [
    RCurl
  ];
};

Here we define our package set. This will just give us access to all of the Emacs packages defined in Nixpkgs.

customEmacsPackages = emacsPackagesNg.overrideScope (super: self: {
  inherit emacs;
});

Next, we define aspell with the English language. This is used by Emacs ispell.

myAspell = aspellWithDicts (ps : with ps; [ en ]);

Tex live provides some LaTeX commads for us.

myTex = texlive.combine {
  inherit (texlive) xetex setspace
    fontspec chktex enumitem xifthen
    ifmtarg filehook wrapfig inconsolata
    upquote minted lastpage collection-basic
    collection-binextra collection-context
    collection-fontsrecommended collection-fontutils
    collection-genericrecommended collection-langenglish
    collection-latex collection-latexrecommended collection-luatex
    collection-mathextra collection-metapost collection-plainextra
    collection-texworks collection-xetex capt-of;
};

Emacs configuration

Here, we start building up the site-paths.el file. This does a simple substitution of all the attributes set.

  site-paths = runCommand "site-paths.el" {
    inherit rtags ripgrep ag emacs ant nethack fortune gnutls
      coreutils findutils openssh git bash
      zsh perl golint perlcritic
      go asciidoc lessc stack
      lua gcc bashInteractive diffutils
      pandoc clang cmake ghostscript
      gnugrep man gawk gnused
      sqliteInteractive freetds mariadb
      parallel unixODBC ncompress
      texinfoInteractive cacert notmuch;
    inherit (pythonPackages) flake8;
    inherit (nodePackages) jshint eslint;
    intero = haskellPackages.intero-nix-shim;
    texlive = myTex;
    markdown2 = pythonPackages.markdown2;
    tidy = html-tidy;
    irony = irony-server;
    libxml2 = libxml2.bin;
    gpg = gnupg1compat;
    # jdeeserver = jdee-server;
    aspell = myAspell;
    orginit = ./org-init.el;
  } ''
cp ${./site-paths.el.in} $out
substituteAllInPlace $out
  '';

Emacs building can be divided into phases. Each phase will run through the Elisp once.

Phase 1: picking up dependencies

myPackages gets a listing of all of the packages that are needed by the Emacs configuration. use-package-list generates this list automatically.

  package-list = with customEmacsPackages.melpaPackages;
    runCommand "package-list" { buildInputs = [ emacs ]; } ''
emacs --batch --quick \
                -L ${bind-key}/share/emacs/site-lisp/elpa/bind-key-* \
                -L ${use-package}/share/emacs/site-lisp/elpa/use-package-* \
                -L ${./lisp} -l use-package-list \
                --eval "(use-package-list \"${./README.el}\")" > $out
  '';
  myPackages = builtins.fromJSON (builtins.readFile package-list);

Phase 2: byte compiling

  default = runCommand "default.el" { buildInputs = [emacs]; } ''
mkdir -p $out/share/emacs/site-lisp

cp ${./README.el} $out/share/emacs/site-lisp/default.el
cp ${site-paths} $out/share/emacs/site-lisp/site-paths.el

cp ${./lisp/em-dired.el} $out/share/emacs/site-lisp/em-dired.el
cp ${./lisp/dired-column.el} \
  $out/share/emacs/site-lisp/dired-column.el
cp ${./lisp/macho-mode.el} $out/share/emacs/site-lisp/macho-mode.el
cp ${./lisp/nethack.el} $out/share/emacs/site-lisp/nethack.el
cp ${./lisp/set-defaults.el} \
  $out/share/emacs/site-lisp/set-defaults.el
cp ${./lisp/installer.el} $out/share/emacs/site-lisp/installer.el
cp ${./lisp/restart-emacs.el} \
  $out/share/emacs/site-lisp/restart-emacs.el
cp ${./lisp/company-eshell-history.el} \
  $out/share/emacs/site-lisp/company-eshell-history.el
cp ${./lisp/use-package-list.el} \
  $out/share/emacs/site-lisp/use-package-list.el

Ideally, we could compile the .el file into a .elc, but it doesn’t work so well. The below block should do it, but it gives lots of errors.

cd $out/share/emacs/site-lisp
export HOME=$PWD
emacs --batch --quick \
      --eval "(let ((default-directory \"${emacsWrapper
                ((requiredPackages customEmacsPackages myPackages) ++
                  [customEmacsPackages.melpaPackages.use-package])
}/share/emacs/site-lisp\")) (normal-top-level-add-subdirs-to-load-path))" \
      -L . -f batch-byte-compile *.el

'';

Phase 3: wrapping into Emacs

emacsWrapper = explicitRequires: runCommand "emacs-packages-deps"
 { inherit emacs explicitRequires; inherit (xorg) lndir; }
 ''
   mkdir -p $out/bin
   mkdir -p $out/share/emacs/site-lisp
   local requires
   for pkg in $explicitRequires; do
     findInputs $pkg requires propagated-user-env-packages
   done
   linkPath() {
     local pkg=$1
     local origin_path=$2
     local dest_path=$3
     # Add the path to the search path list, but only if it exists
     if [[ -d "$pkg/$origin_path" ]]; then
       $lndir/bin/lndir -silent "$pkg/$origin_path" "$out/$dest_path"
     fi
   }
   linkEmacsPackage() {
     linkPath "$1" "bin" "bin"
     linkPath "$1" "share/emacs/site-lisp" "share/emacs/site-lisp"
   }
   # Iterate over the array of inputs (avoiding nix's own interpolation)
   for pkg in "''${requires[@]}"; do
     linkEmacsPackage $pkg
   done
   # Byte-compiling improves start-up time only slightly, but costs nothing.
   $emacs/bin/emacs --batch -f batch-byte-compile "$siteStart"
'';

requiredPackages is a function that takes two arguments.

requiredPackages = epkgs: map (x:
  if builtins.hasAttr x epkgs.melpaPackages
    then builtins.getAttr x epkgs.melpaPackages
    else if builtins.hasAttr x epkgs then builtins.getAttr x epkgs
    else if builtins.hasAttr x emacsPackages
    then builtins.getAttr x emacsPackages
    else if builtins.hasAttr x epkgs.elpaPackages
    then builtins.getAttr x epkgs.elpaPackages
    else builtins.getAttr x pkgs);

Now we build our Emacs distribution.

myEmacs = customEmacsPackages.emacsWithPackages (epkgs:
  (requiredPackages epkgs myPackages)
  ++ [default epkgs.melpaPackages.use-package]
);

The environment

Finally, we can actually build the environment.

myEnv = buildEnv {
  buildInputs = [ makeWrapper ];
  postBuild = ''
if [ -w $out/share/info ]; then
  shopt -s nullglob
  for i in $out/share/info/*.info $out/share/info/*.info.gz; do # */
    ${texinfoInteractive}/bin/install-info $i $out/share/info/dir
  done
fi

mkdir -p $out/etc

cp ${./gitconfig} $out/etc/gitconfig
substituteInPlace $out/etc/gitconfig \
  --replace @gitignore@ ${./gitignore} \
  --replace @gnupg@ ${gnupg1compat}/bin/gpg \
  --replace @out@ $out

cp ${./bashrc.sh} $out/etc/bashrc
substituteInPlace $out/etc/bashrc \
  --replace @fortune@ ${fortune} \
  --replace @out@ $out

cp ${./zshrc.sh} $out/etc/.zshrc
substituteInPlace $out/etc/.zshrc \
  --replace @zsh-autosuggestions@ ${zsh-autosuggestions} \
  --replace @fortune@ ${fortune} \
  --replace @out@ $out
cp $out/etc/.zshrc $out/etc/zshrc

cp ${./etc-profile.sh} $out/etc/profile
substituteInPlace $out/etc/profile \
  --replace @out@ $out

wrapProgram $out/bin/bash \
  --add-flags "--rcfile $out/etc/bashrc"

wrapProgram $out/bin/zsh \
  --set ZDOTDIR $out/etc

cp ${./runemacs.sh} $out/bin/run
substituteInPlace $out/bin/run \
  --replace @out@ $out
chmod +x $out/bin/run
  '';
  meta = {
    priority = -10;
  };
  pathsToLink = [
    "/bin"
    "/etc/profile.d"
    "/etc/bash_completion.d"
    "/etc/ssl"
    "/Applications"
    "/share/doc"
    "/share/man"
    "/share/info"
    "/share/zsh"
    "/share/bash-completion"
    "/share/mime"
  ];

  extraOutputsToInstall = [ "man" "info" "doc" "devdoc" "devman" ];
  name = "bauer";

  paths = [
    myEmacs
    myTex
    rEnv
    (runCommand "my-profile" { buildInputs = [makeWrapper]; } ''
mkdir -p $out/etc/profile.d
cp ${./profile.sh} $out/etc/profile.d/my-profile.sh
substituteInPlace $out/etc/profile.d/my-profile.sh \
  --replace @emacs@ ${myEmacs} \
  --replace @fortune@ ${fortune} \
  --replace @cacert@ ${cacert}
    '')
  ] ++ [bashInteractive zsh coreutils git
        gawk gnused gzip gnutar gnupg1compat xz cacert
  ] ++ [nox nix nix-index nix-repl]
    ++ [nodePackages.tern isync notmuch graphviz]
    ++ [stack ghc];
};

in myEnv

Invoking it

We can build it with nix-build.

nix-build
./result/bin/run

Continuous integration

https://travis-ci.org/matthewbauer/bauer“> ">

We’ll set up Travis support here. We start by configuring .travis.yml.

`.travis.yml`

language: nix

Next we’ll set up nix-build and pass the URL of Nixpkgs with the NIXPKGS.

script:
  - nix-build -Q --argstr nixpkgs-url $NIXPKGS
  - sh info.sh result

Setup the OSs. Sadly no Windows support yet.

git:
  depth: 1
sudo: false
os:
  - linux
  - osx

Setup some values for NIXPKGS variables.

env:
  - NIXPKGS=nixos.org/channels/nixos-17.09/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixpkgs-17.09-darwin/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz
  - NIXPKGS=nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

Configure the matrix…

matrix:
  exclude:
    - os: linux
      env: NIXPKGS=nixos.org/channels/nixpkgs-17.09-darwin/nixexprs.tar.xz
    - os: osx
      env: NIXPKGS=nixos.org/channels/nixos-17.09/nixexprs.tar.xz
    - os: osx
      env: NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz
  allow_failures:
    - env: NIXPKGS=nixos.org/channels/nixos-unstable/nixexprs.tar.xz
    - env: NIXPKGS=nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

Setup the cache.

cache:
  directories:
    - /nix/store

Turn off those annoying Travis notifications.

notifications:
  email: false

Extra

`update.sh`

This is a simple script that I use to make sure I’ve updated the generated files.

#!/bin/sh
emacs --batch \
      -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"

`info.sh`

This file gives us some info on the built derivation. Only arg is optional and should be a path to a Nix store derivation.

#!/bin/sh
out=$1

if [ -z "$out" ]; then
  out=$(nix-build)
fi

if [ -L "$out" ]; then
  out=$(readlink -f $out)
fi

if [ -e $out ]; then
  echo Dependencies:
  du -scl $(nix-store -qR $out) | sort -n
fi

`.gitignore`

These set up some paths for .gitignore that we don’t want getting put in the repo. Start with Emacs/org-mode/LaTeX stuff.

flycheck_*.el
*.elc
*.pdf
*.html
*.tex
*.log
*.aux
*.out
*.toc

Nix-related stuff.

# nix stuff
result
result-*

These are all tangled by README.org.

README.el
bashrc.sh
bauer.nix
zshrc.sh
etc-profile.sh
runemacs.sh
gitconfig
gitignore
default.el
profile.sh
site-paths.el.in
init.el
org-init.el

`config.nix`

This file is just for convenience if you wanted to build with the Nixpkgs in your channel. Having this file available means you can place this repo in ~/.nixpkgs and Nixpkgs will have the package userPackages available.

{
  packageOverrides = pkgs: with pkgs; rec {
    bauer = import ./default.nix { nixpkgs' = pkgs; };
    userPackages = bauer;
  };
}

`org-init.el`

(require 'site-paths nil t)
(require 'use-package)
(setq use-package-always-demand t
      use-package-always-ensure nil)
(use-package tex-site)
(use-package emacs-lisp)
(use-package sh-script)
(use-package org
  :config
  (setq org-export-in-background t
        org-src-fontify-natively t
        org-src-preserve-indentation t
        ;; org-html-htmlize-output-type (quote css)
        org-latex-listings (quote minted))
  (use-package ox-latex)
  (use-package ox-beamer)
  (use-package ox-md)
  (use-package ox-rss))

Nix and Org

Sun, 15 Oct 2017 23:26:00 -0500

This is a guide on getting Org to work with Nix. Specifically, I want to show how you can write Nix expressions within an Org document. This might not seem useful at first, but I think by the end of this guide it will become clear how powerful the combination of these tools are. First, some background on what these tools are for those unfamiliar.

Org was started in 2003 as an extension, or ‘mode’, to the Emacs text editor. Carsten Dominik, frustrated with Emacs builtin outliner, sought to create something that he could use to organize all of his notes and projects in a tree structure. Since then, Org has grown to include features for note-taking, hyperlinking, todo lists, spreadsheets, literate programming, and even exporting to HTML and LaTeX (in fact, this blog is composed in Org). It is the literate programming feature, called Babel, that will be used extensively in this guide. All of Org’s features are written in extensible Lisp that manipulate plain-text ‘.org’ files. Carsten’s creation has become extremely popular among Emacs users and since 2007 has been included in Emacs. The easiest way to start using Org is to download Emacs directly from the GNU website.

Let’s start by creating a new Org file. Once you have Emacs started, you can create a new file by pressing C‑x C‑f. If you are unfamiliar with this notation, that means press and hold the Control key, then press x, then press f. Now we can name the new file at the ‘Find file: ’ prompt. Let’s type ‘demo.org’ here and then press enter. Now we have an empty Org mode file to type into. You can save it at any time with C‑x C‑s.

#+title: Org Demo
#+date: <2017-10-15 Sun>

This is just a plain-text document. Keep this document open because we’ll be
editing this file throughout the tutorial.

Now that you have an Org file setup, make sure you leave Emacs open and we’ll refer back to this file later. Next, some background on Nix.

Nix was started in 2004 as a purely functional package manager. It solves many problems in traditional package management. For instance, multiple versions of a package can be used at the same time, each package gets unique dependencies, bad upgrades can be rolled back, and unused packages can be garbage collected. More information on Nix is available from the Nix homepage. Nix can be installed on both Linux and macOS machines. It is fairly easy to setup, provided you have sudo access. Run the following in a terminal to install Nix,

$ curl https://nixos.org/nix/install | sh

Once Nix has been installed, we can go back to Emacs and start writing to demo.org. Let’s append some documentation to the Org file on what we’ll be building.

This Org file builds the Nix expression for ‘Hello World’.

To put source code in our Org file, we’ll need to know some Org syntax. The directives #+begin_src and #+end_src tell Org that we are writing in another language. nix is the name of the language and hello.nix is the file that Org will write (tangle) the code to.

#+begin_src nix :tangle hello.nix
/* Nix code goes here */

Finally, we can write in Nix’s expression language to create a Hello World package. Nix’s syntax can be confusing to new users so I’ll be as verbose as possible. Almost everything is a function in Nix, so, let’s define some arguments for a Hello World function. Functions with multiple arguments are defined with { a, b, c }: where each argument is a comma separated identifier between { and }. We’ll need stdenv (Nix’s standard environment) and fetchurl (Nix’s function to download a URL) to build Hello World.

{ stdenv, fetchurl }:

Next, we need to provide a definition for Hello World. We want to build Hello World from some source files. That’s called a derivation in Nix and we can accomplish it by calling the stdenv.mkDerivation function. The keyword rec tells Nix that our arguments to stdenv.mkDerivation might be recursive, meaning our args might need to reference each other.

stdenv.mkDerivation rec {

Let’s give this derivation a name.

name = "hello-2.10";

Then provide a source using fetchurl.

  src = fetchurl {
    url = "mirror://gnu/hello/${name}.tar.gz";
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
}
#+end_src

That’s it! We’ve defined a Nix expression that will build Hello World.

Unfortunately, this won’t work on it’s own because Nix doesn’t know what stdenv and fetchurl refer to. We’ll need to define another Nix expression to bootstrap this one. Since this is all in Org mode that’s as easy as just adding another code block. We’ll call it bootstrap.nix and it will just call hello.nix as a package using the stdenv and fetchurl defined in Nixpkgs.

#+begin_src nix :tangle bootstrap.nix
{ nixpkgs ?  }:

with import nixpkgs {};

callPackage ./hello.nix {}
#+end_src

Let’s see if this works. Type M‑x (which means hold down the meta key and press x) and then type org‑babel‑tangle‑file followed by enter. After this was run, you’ll have a file called hello.nix and a file called bootstrap.nix in the current directory. Open up a terminal and we can build Hello World.

$ nix-build
/nix/store/…-hello-2.10
$ ./result/bin/hello
Hello World!

This is pretty useful on its own, but I don’t want to have to tangle the file each time I make a change. Let’s tangle Org within a Nix expression. Is this possible in Nix?

Recently Nix has added a feature called IFD which stands for Import From Derivation. It allows us to use generated Nix expressions within another Nix expression. That way we can derive hello.nix from a Nix expression and then derive hello.nix, avoiding org‑babel‑tangle‑file completely.

To do this, we’ll need to create another Nix file that we’ll call default.nix. Like the boostraper, I won’t break down exactly what’s going on in this post, but it follows the basic Import From Derivation process outlined above calling Org’s tangler.

#+begin_src nix :tangle default.nix
{ nixpkgs ?  }:

with import nixpkgs {};

import (runCommand "bootstrap.nix-generated" {
  buildInputs = [ emacs ];
} ''
    mkdir -p $out
    cd $out
    cp ${./test.org} test.org
    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"test.org\")"
    cp bootstrap.nix default.nix
  ''
) { inherit nixpkgs; }
#+end_src

We still need to tangle everything one last time. Type M‑x and then type org‑babel‑tangle‑file followed by enter. A new default.nix file will appear. That default.nix file along with test.org is all you need to build this expression. To verify this, let’s first remove those old files so we know they aren’t being referenced accidentally.

$ rm -f hello.nix bootstrap.nix

Finally, we can build our Hello World expression!

$ nix-build
/nix/store/…-hello-2.10
$ ./result/bin/hello
Hello World!

In this guide I’ve shown you an easy way to write Nix expressions in Org. Now that you have this environment setup, you should try hacking around test.org. You can always run nix‑build to see if a binary builds. Try using other sources, creating a new derivation, adding other dependencies, or hack something else up. I recommend reading through the Nix Pills series for learning Nix and the Org Mode Guide for learning Org.