At work we use docker-compose to run
integration tests on a big project that need to connect to multiple
different databases as well as a few other services. This article is
about how to replace docker-compose by
nix for a local development
environment.
Quick tutorial
nix-shell-fu level 1 lesson
Let's start with a basic shell.nix
example:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
with pkgs: mkShell
{ buildInputs = [ hello ];
shellHook = ''
echo "Using ${hello.name}."
'';
}
And this could be understood in plain English as:
In the packages of nix version 22.11, create a new shell into which
the package hello will be installed. At
the end of the install, run a script that will print the package name.
(Cf digression)
If you copy/paste this in a shell.nix
file and run nix-shell you get:
> nix-shell
nix-shell shell.nix
these 53 paths will be fetched (84.69 MiB download, 524.77 MiB unpacked):
/nix/store/08pckaqznwh0s3822cjp5aji6y1lsm27-libcxx-11.1.0
...
/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13
copying path '/nix/store/49wn01k9yikhjlxc1ym5b6civ29zz3gv-bash-5.1-p16' from 'https://cache.nixos.org'...
...
copying path '/nix/store/4w2rv6s96fwsb4qyw8b9w394010gxriz-stdenv-darwin' from 'https://cache.nixos.org'...
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
If you close the session and run it again, it will be much faster and
will only show this:
❯ nix-shell
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
This is because all dependencies will be cached. OK so, this is level
1 of nix-shell-fu.
Now, let's start level 2.
nix-shell-fu level 2 lesson; scripting and
configuring
This time, we want to launch a full service, as a redis docker would
do. So here is a basic shell script which is similar to the previous one
but will request redis as a dependency
instead of hello and also as a launching
script. From there will add a little bit more features.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
redis-server
'';
}
Again if you run nix-shell here is the result:
❯ nix-shell
these 2 paths will be fetched (2.08 MiB download, 6.99 MiB unpacked):
/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5
/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7
copying path '/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7' from 'https://cache.nixos.org'...
copying path '/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5' from 'https://cache.nixos.org'...
Using redis-7.0.5
97814:C 10 Feb 2023 20:44:36.960 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
97814:C 10 Feb 2023 20:44:36.960 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=97814, just started
97814:C 10 Feb 2023 20:44:36.960 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
97814:M 10 Feb 2023 20:44:36.961 * Increased maximum number of open files to 10032 (it was originally set to 256).
97814:M 10 Feb 2023 20:44:36.961 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 97814
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
97814:M 10 Feb 2023 20:44:36.962 # WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.
97814:M 10 Feb 2023 20:44:36.962 # Server initialized
97814:M 10 Feb 2023 20:44:36.963 * Ready to accept connections
Woo! Redis is started and it works!
But if you have multiple projects you want to have more control. For
example, we will want to run redis on a specific port. Here is how you
do it:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/21.05.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port ${port}"
redis-server --port ${port}
'';
}
And here is the result:
> rm dump.rdb
> nix-shell
Using redis-6.2.3 on port 16380
1785:C 10 Feb 2023 20:50:00.880 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1785:C 10 Feb 2023 20:50:00.880 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=1785, just started
1785:C 10 Feb 2023 20:50:00.880 # Configuration loaded
1785:M 10 Feb 2023 20:50:00.880 * Increased maximum number of open files to 10032 (it was originally set to 256).
1785:M 10 Feb 2023 20:50:00.880 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.3 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 16380
| `-._ `._ / _.-' | PID: 1785
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1785:M 10 Feb 2023 20:50:00.881 # Server initialized
1785:M 10 Feb 2023 20:50:00.881 * Ready to accept connections
Woo! We control the port from the file. That's nice.
But, has you might have noticed, when you quit the session it dumps
the DB as the file dump.rdb. What we would
like is to keep all the state in a local directory that would be easy to
delete.
To achieve this, instead of passing argument to the redis command
line we will use a local config file to use.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell (rec {
# ENV Variables the directory to put all the DATA
REDIS_DATA = "${toString ./.}/.redis";
# the config file, as we use REDIS_DATA variable we just declared in the
# same nix set, we need to use rec
redisConf = pkgs.writeText "redis.conf"
''
port ${port}
dbfilename redis.db
dir ${REDIS_DATA}
'';
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
[ ! -d $REDIS_DATA ] \
&& mkdir -p $REDIS_DATA
cat "$redisConf" > $REDIS_DATA/redis.conf
alias redisstop="echo 'Stopping Redis'; redis-cli -p ${port} shutdown; rm -rf $REDIS_DATA"
nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1 &
echo "When finished just run redisstop && exit"
trap redisstop EXIT
'';
})
And here is a full session using this shell.nix:
> nix-shell
Using redis-6.2.3 on port: 16380
When finished just run redisstop && exit
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380
127.0.0.1:16380> help
redis-cli 6.2.3
To get help about Redis commands type:
"help @<group>" to get a list of commands in <group>
"help <command>" for help on <command>
"help <tab>" to get a list of possible help topics
"quit" to exit
To set redis-cli preferences:
":set hints" enable online hints
":set nohints" disable online hints
Set your preferences in ~/.redisclirc
127.0.0.1:16380>
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. .redis shell.nix
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380 shutdown
[1]+ Done nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.db
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redisstop
Stopping Redis
Could not connect to Redis at 127.0.0.1:16380: Connection refused
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. shell.nix
So with this version all data related to redis is saved into the
local .redis directory. And in the nix
shell we provide a command redisstop that
once invoked, shutdown redis, then purge all redis related data (as you
would like in a development environment). Also, as compared to previous
version, redis is launched in background so you could run commands in
your nix shell.
Notice I also run redisstop command on exit of the
nix-shell. So when you close the nix-shell redis is stopped and the DB
state is cleaned up.
Composable nix-shell
As a quick recap you now have a boilerplate to create new shell.nix:
{ pkgs ? import ( ... ) {} }:
mkShell { MY_ENV_VAR_1 = ...;
MY_ENV_VAR_2 = ...;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' command_to_run_after_init '';
}
But if I give you two such shell.nix
files, would you be able to compose them? Unfortunately, not directly.
To solve the problem we will replace this boilerplate by another one
that do not directly uses mkShell. And in
order to make it fully composable, we will also need to narrow the
environment variables declaration in a sub field:
{ pkgs ? import ( ... ) {} }:
let env = { PGDATA = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' some_command $PG_DATA '';
}
With this, we can compose two nix set into a single merged one that
will be suitable to pass as argument to mkShell. Another
minor detail, but important one. In bash, the command trap
do not accumulate but replace the function. For our need, we want to run
all stop function on exit. So the trap directive added in
the shell hook does not compose naturally. This is why we add a stop value that will contain the name of the
bash function to call to stop and cleanup a service.
Finally the main structure for each of our service will look like
this nix service boilerplate:
{ pkgs ? import ( ... ) {} }:
let env = { MY_SERVICE_ENV_VAR = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' my_command $MY_SERVICE_ENV_VAR '';
stop = "stop_my_service"
}
So let's start easy. To run a single shell script like this with
nix-shell, you should put your service
specific nix file in a service.nix file
and create a shell.nix file that contains
something like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let service = import ./service.nix { inherit pkgs; };
in with service; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})
Now, if you would like to run nix shell for multiple files, here is a
first qui solution:
{ pkgs ? import (...) {}}:
let
# merge all the env sets
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# merge all the confs by accumulating the dependencies
# and concatenating the shell hooks.
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
emptyConf
confs;
in mkShell (mergedEnvs // mergedConfs)
And now, here is the full solution that also deal with other minor
details like importing the files and dealing with the exit of the
shell:
{ mergeShellConfs =
# imports should contain a list of nix files
{ pkgs, imports }:
let confs = map (f: import f { inherit pkgs; }) imports;
envs = map ({env ? {}, ...}: env) confs;
# list the name of a command to stop a service (if none provided just use ':' which mean noop)
stops = map ({stop ? ":", ...}: stop) confs;
# we want to stop all services on exit
stopCmd = builtins.concatStringsSep " && " stops;
# we would like to add a shellHook to cleanup the service that will call
# all cleaning-up function declared in sub-shells
lastConf =
{ shellHook = ''
stopall() { ${stopCmd}; }
echo "You can manually stop all services by calling stopall"
trap stopall EXIT
'';
};
# merge Environment variables needed for other shell environments
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# zeroConf is the minimal empty configuration needed
zeroConf = {buildInputs = []; nativeBuildInputs = []; shellHook="";};
# merge all confs by appending buildInputs and nativeBuildInputs
# and by concatenating the shellHooks
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
zeroConf
(confs ++ [lastConf]);
in (mergedEnvs // mergedConfs);
}
So I put this function declaration in a file named ./nix/merge-shell.nix. And I have a pg.nix as well as a redis.nix file in the nix directory. On the root of the project the
main shell.nix looks like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let
# we import the file, and rename the function mergeShellConfs as mergeShells
mergeShells = (import ./nix/merge-shell.nix).mergeShellConfs;
# we call mergeShells
mergedShellConfs =
mergeShells { inherit pkgs;
# imports = [ ./nix/pg.nix ./nix/redis.nix ];
imports = [ ./nix/pg.nix ./nix/redis.nix ];
};
in pkgs.mkShell mergedShellConfs
And, that's it. Now when I run nix-shell it launch both Postgresql and Redis,
and when I quit the shell, the state is cleaned up. Both postgres and
redis are shutdown and the local files are erased.
I hope this could be useful to someone else.
Appendix
Digression
In fact, this is a bit more complex than "just that". The reality is
a bit more complex. The nix language is "pure", meaning, if you run the
nix evaluation multiple times, it will always evaluate to the exact same
value. But here, this block represent a function. The function takes as
input a "nix set" (which you can see as an associative array, or a
hash-map or also a javascript object depending on your preference), and
this set is expected to contain a field named pkgs. If pkgs is
not provided, it will use the set from the stable version 22.11 of
nixpkgs by downloading them from github archive. The second part of the
function generate "something" that is returned by an internal function
of the standard library provided by nix
which is named mkShell. So mainly, mkShell is a helper function that will generate
what nix calls a derivation.
Mainly, we don't really care about exactly what is a
derivation. This is an internal to nix representation that
could be finally used by different nix tools for different things.
Typically, installing a package, running a local development environment
with nix-shell or nix develop, etc…
So the important detail to remember is that we can manipulate the
parameter we pass to the functions derivation, mkDerivation and mkShell, but we have no mechanism to manipulate
directly derivation. So in order to make
that composable, you need to call the derivation internal function at the very end
only.
The argument of all these functions are nix sets
The full nix files for
postgres
For postgres:
{ pkgs }:
let iport = 15432;
port = toString iport;
pguser = "pguser";
pgpass = "pgpass";
pgdb = "iroh";
# env should contain all variable you need to configure correctly mkShell
# so ENV_VAR, but also any other kind of variables.
env = {
postgresConf =
pkgs.writeText "postgresql.conf"
''
# Add Custom Settings
log_min_messages = warning
log_min_error_statement = error
log_min_duration_statement = 100 # ms
log_connections = on
log_disconnections = on
log_duration = on
#log_line_prefix = '[] '
log_timezone = 'UTC'
log_statement = 'all'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
logging_collector = on
log_min_error_statement = error
'';
postgresInitScript =
pkgs.writeText "init.sql"
''
CREATE DATABASE ${pgdb};
CREATE USER ${pguser} WITH ENCRYPTED PASSWORD '${pgpass}';
GRANT ALL PRIVILEGES ON DATABASE ${pgdb} TO ${pguser};
'';
PGDATA = "${toString ./.}/.pg";
};
in env // {
# Warning if you add an attribute like an ENV VAR you must do it via env.
inherit env;
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.coreutils
pkgs.jdk11
pkgs.lsof
pkgs.plantuml
pkgs.leiningen
];
nativeBuildInputs = [
pkgs.zsh
pkgs.vim
pkgs.nixpkgs-fmt
pkgs.postgresql_11
# postgres-11 with postgis support
# (pkgs.postgresql_11.withPackages (p: [ p.postgis ]))
];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.postgresql_11.name}. port: ${port} user: ${pguser} pass: ${pgpass}"
# Setup: other env variables
export PGHOST="$PGDATA"
# Setup: DB
[ ! -d $PGDATA ] \
&& pg_ctl initdb -o "-U postgres" \
&& cat "$postgresConf" >> $PGDATA/postgresql.conf
pg_ctl -o "-p ${port} -k $PGDATA" start
echo "Creating DB and User"
psql -U postgres -p ${port} -f $postgresInitScript
function pgstop {
echo "Stopping and Cleaning up Postgres";
pg_ctl stop && rm -rf $PGDATA
}
alias pg="psql -p ${port} -U postgres"
echo "Send SQL commands with pg"
trap pgstop EXIT
'';
stop = "pgstop";
}
And to just launch Posgresql, there is also this file ./nix/pgshell.nix, that simply contains
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let pg = import ./pg.nix { inherit pkgs; };
in with pg; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})