Matej Cotman's Blog - NixOS

Multi VPN setup

Sun, 20 Nov 2022 00:00:00 +0100

Running multiple unprivileged OpenVPN instances at the same time with every network having separated set of applications.

Goals

I never liked how VPNs are integrating into computer’s main network. But what if I tell you we can separate that and do even more:

Let’s do this:

Multiple VPNs at the same time
Do not touch primary computer networking
Run specific applications per specific VPN
Do all this without privileged access

Linux Namespaces

We can do all above with setting up Linux Namespaces which is a Linux kernel feature and is already in stable versions of kernels for some time.

What are Linux Namespaces?

Feature of Linux kernel that partitions resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources.

Let’s see what kind of resources are available to partition:

Mount
Process ID
Network
Inter-process communication
UTS (hostname)
User
Control group
Time

Note: This is how containerization works and this is why you need some sort of virtual machine running Linux when using Docker on non Linux operating systems.

Let’s see the solution

Although there are many ways to set up this kind of solution, I will be using:

Namespace setup tool: nsjail
User mode networking: slirp4netns
To bind the things together: Nix

With nsjail we will be setting up namespaces and run the OpenVPN inside along with our predefined graphical applications; for process manager inside the nsjail I choose SupervisorD. Slirp4netns will be providing user space network. While Nix will take care of all dependencies.

Visualization of one nsjail instance:

Usage

First, we require standalone nsjail.nix file which resides in my playground repository named helper_scripts. We will extend that file as many times as we have OpenVPN environments.

The Nix script (extended from nsjail.nix)

Let us save this file as myvpn.nix.

{ pkgs ? import <nixpkgs> {} }:
import /home/matejc/workarea/helper_scripts/nixes/vpn/nsjail.nix rec {
  inherit pkgs;
  name = "myvpn";
  user = "matejc";
  homeDir = "/home/${user}";
  vpnStart = "openvpn --config /etc/openvpn/ovpn --daemon --auth-user-pass /etc/openvpn/pass";
  vpnStop = "pkill openvpn";
  openvpnConfig = null;
  cmds = [
    { start = "firefox --no-remote"; }
  ];
  packages = [ pkgs.firefox ];
  romounts = [
    { from = "/run/opengl-driver"; to = "/run/opengl-driver"; }
    { from = "${homeDir}/.vpn/${name}/etc/openvpn"; to = "/etc/openvpn"; }
  ];
}

Parameters

pkgs

Pass nixpkgs to the nsjail.nix.

name

Name of the environments, need to be unique since the state path is created like so:

${homeDir}/.vpn/${name}/

Example:

/home/matejc/.vpn/myvpn/

user

Username of user running this environment.

homeDir

Home dir of user running this environment.

vpnStart

Command to run OpenVPN (must be in daemon mode).

vpnStop

Command to stop OpenVPN.

openvpnConfig

Main configuration file for OpenVPN, if this is set, you do not need vpnStart and vpnStop, but then there is no way to set extra OpenVPN arguments like –auth-user-pass, so we set it to null.

cmds

Commands to start after OpenVPN is connected.

packages

Nix packages used in cmds.

romounts

Read-only mounts.

Here we mount OpenGL drivers and openvpn folder.

Running

Before we run it first time, we need to add ovpn (OpenVPN connection configuration file) and pass (credentials) file to the proper path: ${homeDir}/.vpn/${name}/etc/openvpn.

Example:

mkdir -p $HOME/.vpn/myvpn/etc/openvpn
cp config.ovpn $HOME/.vpn/myvpn/etc/openvpn/ovpn
printf "username\npassword" > $HOME/.vpn/myvpn/etc/openvpn/pass

Now, to run Firefox inside OpenVPN network, you just need to execute: nix-shell ./myvpn.nix. Soon the Firefox should appear on your desktop, you can check in this Firefox which public IP you have (for example, navigate to https://www.ipaddress.my/) and compare it with the browser you normally use.

To quit the environment (exit from the browser and kill the OpenVPN) you need to type Ctrl+C for the nix-shell command.

Conclusion

I am using this setup on NixOS, in the past I used it on Fedora Silverblue with Nix installed, but it requires a bit different configuration (talking about the myvpn.nix file).

Why is this useful, you might ask? Well this is more like a prototype, so it can be rough on the edges, but even now can be useful when you would like to retain your network while accessing some web service through VPN. One use case is, you work as a consultant at a company A and having a client B as one does, you could use your primary network for company A and access services of company B through this solution.

Neovim, WSL and Nix

Sat, 08 Aug 2020 00:00:00 +0200

How to use Neovim (Neovim-Qt) under WSL 1/2 with the power of Nix

Intro

Well we all know that generally development on Linux is easier than on Windows, but sometimes you are forced to use Windows. But that does not mean that all those nice tools from Linux are not available to you, as we will see in this post.

Windows has some thing called WSL which enables you to run Linux tools natively in the Windows subsystem. Not all is without issues, you can not run graphical Linux applications because Windows does not run Xorg server, yeah you have Xorg ports that run there but that is in this case just one more unwanted layer, remember, building efficient solutions is what every engineer should strive to.

What I did is to use Windows pre-built binaries of Neovim-Qt and run the Neovim installed with Nix inside WSL.

Ok, you could say then, why not use VS Code with some Vim/Neovim plugin and use so called Remote-WSL plugin to access WSL… Well yes, but at least me I stumble upon few issues. First was that CPU usage was through the roof when Remote-WSL extension was in use on WSL1 (I could not just run Windows Update on client’s managed computer) and the fix was to install specific version of libc with dpkg (which is absurd in the first place because this is a good way to ruin your whole environment). Applying this fix did the trick for lowering the CPU usage. The second issue come right after, when I wanted to install some package with APT package manager, like I predicted, libc install did its damage, I could not install or un-install anything with APT. Nix comes again to the rescue.

By the way the sleep command forgot how to work under WSL and Ubuntu 20.04 Source.

Let’s see the solution

Neovim-Qt

Neovim-Qt has nicely built binaries on their GitHub page for Windows, so I just downloaded that zip and unpacked it into C:/Program Files/neovim-qt/. But any location could do.

WSL

Open PowerShell as Administrator and run:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

If you do not have up to date Windows for any kind of reason to install WLS2 then reboot now.

Reboot Time

You should have now enabled the WSL1 and you can proceed to install Ubuntu 20.04 (or any other Linux distro you like) from Microsoft store. Do not forget to click Launch after installing it (it will ask you to create a user).

Nix

To install Nix, you need to first open some terminal emulator and run wsl.exe, but you can also just run it from Start menu.

bash <(curl -L https://nixos.org/nix/install)

To finish you can just close the terminal and open wsl.exe again.

Thats it.

The Nix script

Now here is the absolutely most awesome part that connects everything together.

{ pkgs ? import <nixpkgs> {} }:
pkgs.writeScript "run-neovim-qt.sh" ''
    #!${pkgs.stdenv.shell}
    set -e

    # get random free port
    export NVIM_LISTEN="127.0.0.1:$(${pkgs.python3Packages.python}/bin/python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')"

    # use python's sleep, because coreutils' sleep does not function under Ubuntu 20.04 and WSL
    #   after delay start nvim-qt - so that nvim starts before the GUI
    { ${pkgs.python3Packages.python}/bin/python -c 'import time; time.sleep(1)'; "''${NVIM_QT_PATH}" --server "$NVIM_LISTEN"; } &

    # start nvim
    ${pkgs.neovim}/bin/nvim --listen "$NVIM_LISTEN" --headless "$@"
''

Save it to your drive or download with wget under WSL:

wget https://raw.githubusercontent.com/matejc/helper_scripts/master/nixes/neovim-qt.nix

Then build the command with:

nix-build ./neovim-qt.nix

The resulting script is ./result.

Usage

First we need to tell the script where is the Neovim-qt located:

export NVIM_QT_PATH='/mnt/c/Program Files/neovim-qt/bin/nvim-qt.exe'

You can save this into .bashrc or .profile and restart the terminal so that you do not need to repeat the step every time you run wsl shell.

The final step is:

./result my/awesome/code.py

Conclusion

Too much work? You think? Well how much more time you would use using and configuring VS Code or Atom to work under similar environment? And what about Nix? You can install it without the use of native package managers (in case the native one is b0rked) and once you do, you have the power to install your favorite development environment with single command.

I like this solution, in my eyes its simple and efficient, what are your thoughts?

Until next time… I wish you happy hacking!

Links

High cpu usage of node process in Remote-WSL extension #2921 Neovim-Qt Releases WSL on Windows 10 Quick start with Nix

Graphical UI for Nix/NixOS

Mon, 26 Jan 2015 00:00:00 +0100

I started this project as a web UI for Nix, but after my first blog post I had a lot of questions from you, what about the desktop application? So here it is.

What is new?

rewritten app from web UI to desktop UI
still JavaScript, more accurately NodeWebkit
beside package manager there is now configuration browser
no more services, like web server and ElasticSearch
and still using Polymer

Preview of 0.1.1 version

Front Page - Package Manager

Front Page - Configuration Options

Package Browser

Package Info

Manage Marked

Configuration Browser

How to try it out?

There are two ways, the development version and from the nixpkgs. Both described below.

Development

To get the code you will need git:

$ git clone https://github.com/matejc/nixui
$ cd nixui

If you do NOT use

1	nix-channel

to update, then modify config file

1
./src/config.json

so that

1
NIX_PATH

entry has

1
nixpkgs=/path/to/your/nixpkgs

Mine looks like this:

{
    "profilePaths": ["/nix/var/nix/profiles"],
    "dataDir": "/tmp",
    "configurations": ["/etc/nixos/configuration.nix"],
    "NIX_PATH": "nixpkgs=/home/matej/workarea/nixpkgs:nixos=/home/matej/workarea/nixpkgs/nixos:nixos-config=/etc/nixos/configuration.nix:services=/etc/nixos/services"
}

Then build and run NixUI with following command, you will need

gnumake

and

1
nix

for this:

$ make just-run-it

Nixpkgs

Again, if you do NOT use

1	nix-channel

to update, then make sure that

1
config.nixui.NIX_PATH

entry has

1
nixpkgs=/path/to/your/nixpkgs

. In your main configuration nix file add the line

1
nixpkgs.config.nixui.NIX_PATH="nixpkgs=/path/to/your/nixpkgs";

, OR in user config add this line to

1
~/.nixpkgs/config.nix

1
nixui.NIX_PATH="nixpkgs=/path/to/your/nixpkgs";

Install it like any other package, like so:

$ nix-env -iA pkgs.nixui

Warning: at the time of writing this blog post, the

nixui

package was NOT yet in unstable channel, but was merged into master, therefore it is only a matter of time when it will be available.

Plans for the future

In the near future, bug fixes and after NixUI gets stable enough and I get some money from other projects, I will start on Nix configuration editing.

Too many Nix pills, I am addicted

Sat, 20 Dec 2014 00:00:00 +0100

In this blog post you are going to see how to make simple determenistic application with the power of Nix.

First of all, no offence to Luca Bruno aka Lethalman - the inventor of Nix pills

Second of all, do not worry, Nix pills are not actual pills.

There was a joke a month back … about an app that shows system usage of remote computer in your system tray or something like that… well I made it. Khm… apperantly all my apps start with a joke.

Lets start with usage

Get the code from github:

git clone git://github.com/matejc/cgisysinfo.git
cd cgisysinfo

This will build the app to Nix store:

nix-build --argstr prefix `pwd` \
    --argstr listenAddress "0.0.0.0" \
    --argstr listenPort "8080" \
    --argstr user "matejc" \
    --argstr password "mypassword"

To run the app, just execute in current folder:

./result/bin/cgisysinfo-run

To use it on remote machine you have to forward

8080/tcp

port (from example).

To test if it is working on remote machine use following command:

notify-send "`curl --user matejc:<your-password> -k https://<yourip>:8080/`"

Or just open in browser the following page:

1	https://<yourip>:8080/

, the browser should complain about unsecure connection but this is just because the cert is self-signed. After that you will have to enter username and password.

You could try also other scripts .. like

1	https://<yourip>:8080/hello.pl

1
https://<yourip>:8080/hello.py

Code

The program is in one short file, lets go through it:

At the start we declare parameters and its default values:

pkgs: the root of all packages
prefix: where the logs, certs, unix socket, pid files lives
listenAddress: address, where the Nginx will listen
listenPort: port on which Nginx will listen
user: username for simpleauth, used when accessing cgi scripts with curl or something
password: password for simpleauth
templatesFile: the file where you will have templates.nix
extraNginxConf: if you want to add extra Nginx configuration in the 1 server block

1
2
3
4
5
6
7
8
9
{ pkgs ? import <nixpkgs> {}
, prefix ? "/var/lib/cgisysinfo"
, listenAddress ? "localhost"
, listenPort ? "9999"
, user ? "user"
, password ? "password"
, templatesFile ? "${prefix}/templates.nix"
, extraNginxConf ? "" }:
let

Now we have to create a

1	nginx.conf

- the web server configuration with wich we are going serve cgi scripts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
  nginxconf = pkgs.writeText "nginx.conf" ''
  pid ${prefix}/nginx.pid;
  worker_processes 1;
  events {
    worker_connections 128;
  }
  http {
    server {
      access_log ${prefix}/cgi.access.log;
      error_log ${prefix}/cgi.error.log;

      ssl on;
      ssl_certificate     ${prefix}/ssl/selfsigned.crt;
      ssl_certificate_key ${prefix}/ssl/selfsigned.key;

      root ${scripts}/www;
      index index.sh index.pl index.py;
      listen ${listenAddress}:${listenPort};

      location ~ .(py|pl|sh)$ {
        expires -1;

        auth_basic "closed site";
        auth_basic_user_file ${htpasswd};

        gzip           off;
        fastcgi_pass   unix:${prefix}/fcgiwrap.socket;

        # include      fastcgi_params;
        fastcgi_param  QUERY_STRING       $query_string;
        fastcgi_param  REQUEST_METHOD     $request_method;
        fastcgi_param  CONTENT_TYPE       $content_type;
        fastcgi_param  CONTENT_LENGTH     $content_length;

        fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
        fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
        fastcgi_param  REQUEST_URI        $request_uri;
        fastcgi_param  DOCUMENT_URI       $document_uri;
        fastcgi_param  DOCUMENT_ROOT      $document_root;
        fastcgi_param  SERVER_PROTOCOL    $server_protocol;

        fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
        fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

        fastcgi_param  REMOTE_ADDR        $remote_addr;
        fastcgi_param  REMOTE_PORT        $remote_port;
        fastcgi_param  SERVER_ADDR        $server_addr;
        fastcgi_param  SERVER_PORT        $server_port;
        fastcgi_param  SERVER_NAME        $server_name;
      }

      ${extraNginxConf}
    }
  }
  '';

Set variable

1	socketPath

for

1
fcgiwrap.socket

and create file

1
htpasswd

for simple authentication for Nginx (when built, password will be hashed - there will be no plain text password in store).

1
2
3
4
5
6
7
8
9
10
  socketPath = "${prefix}/fcgiwrap.socket";

  htpasswd = pkgs.stdenv.mkDerivation {
    name = "${user}-htpasswd";
    phases = "installPhase";
    installPhase = ''
      export PATH="${pkgs.openssl}/bin:$PATH"
      printf "${user}:$(openssl passwd -crypt ${password})\n" >> $out
    '';
  };

Here comes a bit of magic, we take

1	templatesFile

which we are going to see later and make python/perl/bash scripts from it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
  scripts =
    let
      templates = import templatesFile {inherit pkgs prefix;};
      paths = map (template:
        pkgs.writeTextFile rec {
          name = "cgisysinfo-${template.name}";
          text = template.text;
          executable = true;
          destination = "/www/${template.name}";
        }) templates;
    in
      pkgs.buildEnv {
        name = "cgisysinfo-scripts";
        inherit paths;
        pathsToLink = [ "/www" ];
      };

Main run script takes care of starting

fcgiwrap

and

1
Nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  run = pkgs.writeScriptBin "cgisysinfo-run" ''
  #!${pkgs.bash}/bin/bash

  # we send QUIT signal to Nginx when you press Ctrl+C
  function stopall() {
    kill -QUIT $( cat ${prefix}/nginx.pid )
    exit
  }
  trap "stopall" INT

  export PATH="${pkgs.fcgiwrap}/sbin:${pkgs.nginx}/bin:${pkgs.openssl}/bin:$PATH"

  # generate self-signed cert for nginx (if folder 'ssl' does not exist yet)
  test -d ${prefix}/ssl || \
    { mkdir -p ${prefix}/ssl && \
    openssl req -new -x509 -nodes -keyout ${prefix}/ssl/selfsigned.key -out ${prefix}/ssl/selfsigned.crt -subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=example.xyz"; }

  test -S ${socketPath} && unlink ${socketPath}

  echo -e "\nExample usage:"
  echo "$ notify-send \"\`curl --user ${user}:<your-password> -k https://${listenAddress}:${listenPort}/\`\""
  echo -e "\nPress Ctrl+C to stop ..."

  # start nginx as daemon
  mkdir -p ${prefix}/var/logs
  nginx -c ${nginxconf} -p ${prefix}/var

  # start fcgiwrap (this one blocks)
  fcgiwrap -c 1 -s unix:${socketPath}
  '';

At the end we install

1	cgisysinfo-run

command to Nix store. I also added a

1
shellHook

to run app with

1
nix-shell

1
2
3
4
5
6
7
8
9
10
11
12
13
14
  cgisysinfo = pkgs.stdenv.mkDerivation rec {
    name = "cgisysinfo-${version}";
    version = "0.2";
    unpackPhase = "true";
    installPhase = ''
      mkdir -p $out/bin
      ln -s ${run}/bin/* $out/bin
    '';
    shellHook = ''
      ${run}/bin/cgisysinfo-run
    '';
  };

in cgisysinfo

1
templatesFile

This is an example of

1	templates.nix

which is also located on github as example. Currently only

1
.py

1
.pl

and

1
.sh

files are supported.

Ok lets make one thing clear at this point, for security reasons, be very carefull what information and how you expose on the internet, this is literally remote code execution app.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
{ pkgs, prefix }:
[

  {
    name = "index.sh";
    text = ''
      #!${pkgs.bash}/bin/bash

      export PATH="$PATH:${pkgs.procps}/bin:${pkgs.sysstat}${pkgs.sysstat}/bin"
      echo -e "Content-type: text/plain\n\n"

      echo RAM: `free -mh | awk 'NR==2{ print $3"/"$2 }'`
      echo Swap: `free -mh | awk 'NR==3{ print $3"/"$2 }'`
      ps -eo pcpu,pmem,user,args | sort -k 1 -r | awk 'NR>1 && NR<5{n=split($4,a,"/"); print a[n]": cpu:"$1"%, mem:"$2"%, u:"$3}'
      echo
    '';
  }

  {
    name = "hello.pl";
    text = ''
      #!${pkgs.perl}/bin/perl

      print "Content-type: text/html\n\n";
      print "<html><body>Hello, Perl world.</body></html>";
    '';
  }

  {
    name = "hello.py";
    text = ''
      #!${pkgs.python27}/bin/python

      print "Content-type: text/html\n\n";
      print "<html><body>Hello, Python world.</body></html>";
    '';
  }

]

You have seen how easy is to make a Nix application. This is it. Happy hacking!

Generate package info

Tue, 25 Nov 2014 00:00:00 +0100

Generate Nix packages information with Nix expressions

Lets go right to it!

System requirement is Nix.

Save this Nix expression code to

1	packages.nix

or some other file, every function is commented with examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
let
  # set allowBroken and allowUnfree to true, so that we minimize error output later on
  pkgs = import <nixpkgs> { config = { allowBroken = true; allowUnfree = true; }; };

  # catch exceptions on isDerivation function
  tryDrv = a: builtins.tryEval (pkgs.lib.isDerivation a);
  isDerivation = a: let t = tryDrv a; in t.success && t.value == true;

  # catch exceptions on isAttrs function
  tryAttrs = a: builtins.tryEval (pkgs.lib.isAttrs a);
  isAttrs = a: let t = tryAttrs a; in t.success && t.value == true;

  # iterate through attributeset's names (one-level deep)
  # example:
  # mapValues (name: value: name) pkgs
  # => [ "bash" "zsh" "gitFull" ... ]
  mapValues = f: set: (
    map (attr: f attr (builtins.getAttr attr set)) (builtins.attrNames set)
  );

  # recurse into attributeset (search for derivations)
  # example #1:
  # mapAttrsRecursiveDrv
  #   (path: value: path) pkgs.pythonPackages ["pkgs" "pythonPackages"] []
  # => [ [ "pkgs" "pythonPackages" "searx" ] [ "pkgs" "pythonPackages" "tarman" ] ... ]
  # example #2:
  # mapAttrsRecursiveDrv (path: value: path) pkgs ["pkgs"] []
  # => [ [ "pkgs" "bash" ] [ "pkgs" "zsh" ] [ "pkgs" "pythonPackages" "searx" ] [ "pkgs" "pythonPackages" "tarman" ] ... ]
  mapAttrsRecursiveDrv = f: set: path: list:
    let
      recurse = path: set: visitList:
        let
          visitedFun = a: path:
            let
              isAtt = isAttrs a;
              isDrv = isDerivation a;
              success = if isAtt && !isDrv then pkgs.lib.any (
                element: element == a
              ) visitList else false;
              not = !success;
              list = if not then (visitList ++ [a]) else visitList;
            in
              { inherit list not isAtt isDrv; };

          g = name: value:
            let
              visited = visitedFun value path;
            in
            if visited.isDrv then
              f (path ++ [name]) value
            else if (visited.not) && (checkForEnterable value) then
              recurse (path ++ [name]) value visited.list
            else
              {
                error = "not derivation or not enterable";
                attrPath = pkgs.lib.concatStringsSep "." (path ++ [name]);
              };
        in mapValues g set;
    in (recurse path set list);

  # check if attributeste has attribute named "recurseForDerivations"
  #   therefore has derivations
  # examples:
  # checkForEnterable pkgs.bash => false
  # checkForEnterable pkgs.pythonPackages => true
  checkForEnterable = a:
    let
      t = builtins.tryEval (
        (pkgs.lib.isAttrs a) &&
        (pkgs.lib.hasAttr "recurseForDerivations" a)
      );
    in
      (t.success && t.value == true);

  # main function
  # example:
  # recurseInto "pkgs.pythonPackages"
  # => [
  #   { attrPath = "pkgs.pythonPackages.tarman"; name = "python2.7-tarman-0.1.3"; out = "/nix/store/<hash>-python2.7-tarman-0.1.3"; }
  #   { attrPath = "pkgs.pythonPackages.searx"; name = "python2.7-searx-dev"; out = "/nix/store/<hash>-python2.7-searx-dev"; }
  #   { attrPath = "pkgs.pythonPackages.isPy27"; error = "not derivation or not enterable"; }
  # ]
  recurseInto = attrPath:
    let
      path = pkgs.lib.splitString "." attrPath;
      attrs = pkgs.lib.getAttrFromPath path pkgs;
    in
      pkgs.lib.flatten (mapAttrsRecursiveDrv
        (path: value:
          let
            attrPath = pkgs.lib.concatStringsSep "." path;
            tOutPath = builtins.tryEval value.outPath;
            tName = builtins.tryEval value.name;
          in
            (if tOutPath.success && tName.success then
              { out = tOutPath.value; name = tName.value; inherit attrPath; }
            else
              { error = "tryEval failed"; inherit attrPath; })
        )
        attrs
        path
        []);

  # just strips away values with attribute "error"
  removeErrors = builtins.filter (x: (if pkgs.lib.hasAttr "error" x then
      (builtins.trace "error '${x.error}' at attribute ${x.attrPath}" false)
    else true));

in
  removeErrors (recurseInto "pkgs")

And run it as:

$ nix-instantiate packages.nix --eval --strict --show-trace

It should take up to 15 seconds (well that depends on your system).

Output

Output should look like this, but much more of it (around 3MB if you redirect the stdout to file):

[
    {
        attrPath = "pkgs.pythonPackages.tarman";
        name = "python2.7-tarman-0.1.3";
        out = "/nix/store/<hash>-python2.7-tarman-0.1.3";
    } {
        attrPath = "pkgs.pythonPackages.searx";
        name = "python2.7-searx-dev";
        out = "/nix/store/<hash>-python2.7-searx-dev";
    } {
        attrPath = "pkgs.pythonPackages.isPy27";
        error = "not derivation or not enterable";
    }
    .
    .
    .
]

Every valid item in list has

attrPath

which represent attribute path in

1
pkgs

structure.

If there are some errors, they will be in

error

attribute, always beside

1
attrPath

. There are two error messages currently:

1
not derivation or not enterable
: when attribute set can not be recursable further.
1
tryEval failed
: when there is eval error on attribute.outPath or attribute.name.

And if there is no error for that package, following attributes are available:

1
attrPath
: path, with 1 . seperators, this can be used for package id (this never changes per package).
1
name
: package name with version.
1
out
: out path, this can be used as as unique id further on (can be different per same version of package).

Conclusion

The last line of

1	packages.nix

code can be changed from

removeErrors (recurseInto "pkgs")

builtins.toJSON (removeErrors (recurseInto "pkgs"))

to get JSON encoded string to feed data to some other application.

If you need some other info beside

attrPath

1
name

and

1
out

, just add attributes around the line

1
96

. For example if you want to add

1
meta

attribute add

1
meta = value.meta;

like so:

{ out = tOutPath.value; name = tName.value; inherit attrPath; meta = value.meta; }

But beware, this can increase output data size correspondly.

This is it …

.. and feel free to leave a comment.

Nix's very own graphical UI

Thu, 07 Aug 2014 00:00:00 +0200

I have been working on this project for the last two months and now it is time for some feedback

I have been using NixOS operating system for about two years now, and it is time that I do something more than to package packages and play with Hydra.

At the last year’s NixOS sprint called Zidanca sprint here in Slovenia I had a little joke about making front-end for Nix package manager in JavaScript, well, now it is not a joke any more.

Why JavaScript?

This language had become quite popular lately so I decided to take a look at it, and I like a new challenge.

Technologies used:

NodeJS for backend (expressjs)
Nix package manager commands (nix-env and nix-instantiate)
Web Components for frontend (Polymer)
ElasticSearch for search and package cache

Well, let’s start…

There is no npm/nix package made yet, but I am planing to do it when the project reaches right maturity, should be soon.

Requirements:

git
make
Nix package manager
Browser

Code is at GitHub so let’s get it and enter the folder

$ git clone https://github.com/matejc/nixui
$ cd nixui

To build the environment (get node, bower and elasticsearch packages) and place links to the right places inside the folder, do not worry this is not going to touch your system

$ make build

This command fires up Nix package manager and installs all the packages in custom environment (make is here just so that you do not have to write a bunch of commands and make appropriate links)

This is going to take a few minutes and when it is done, you should have bin folder (well, a link to it), next, you will have to run ElasticSearch:

$ ./bin/services-nixui-start-services

you can check if it went well with this:

$ cat ./var/log/elasticsearch-*.log

There is only one more step, run the server

$ make develop

If everything went well, you will see the message (wait a few seconds to appear, it has to index all the packages):

1	NixUI at http://localhost:8000/index.html

Success! Open the browser at

1	http://localhost:8000/index.html

and you should see something like this

This process was tested on NixOS and should work on any Linux distribution with Nix package manager installed.

Usage

Searching with wildcard

Search for installed packages with

!i

Web UI has simple authentication for security reasons - if you have server running while some other user is running different session - we do not want that others install packages in your environment.

Click on the package and enter development/testing credentials: user: bob, password: secret

On package information window you can mark package for install, click on

1	AVAILABLE!

button or for removal from marked list, click on

1
MARKED AS INSTALLED!

, for unistallation the process is similar (button will appear when you enter credentials).

Manage marked dialog

From the menu (top right) select

1	Manage Marked ...

Here you can apply (install/uninstall) or remove marked packages.

Plans for the future

I will be at the NixOS sprint, well not only that, I am helping to organize it. At the sprint I will be working on your wishes to make this project better.

Managing packages in your user environment is scratching the surface of what you can do with Nix, so I have an idea to visualize and later even edit your main NixOS configuration.nix file as a tree or some other cool structure. Traces of this idea are already in project’s GitHub repository as Nix scripts (src/config_all.nix and src/config_inuse.nix).

Comments and help are much obliged!

And I am available for hire!

This concludes our transmission, I hope you liked it, but stay tuned, project is under development!

Control your packages with Nix created profiles!

Mon, 27 Jan 2014 00:00:00 +0100

Nix has many advantages over most of the other package managers, today, I am going to write about one of them.

Nix is a package manager for the NixOS, but you can install it to other operating systems. One problem on so-called normal systems is that you are/should be very jumpy when installing packages, because you just do not know what and specially where it is going to be installed. Try removing that then…

Well there is a solution - Nix - which works like this: when you install the package, it goes directly to some long-named folder with hash in its name to /nix/store/… and linked into the profile which is available to specific user or all of the users, you can create environments with those profiles.

There are three ways to install a package with Nix

Into the system environment inside global system /etc/nixos/configuration.nix (available only for NixOS users),
into the user environment with command *nix-env –install * and
into the custom environment

System environment

Example (declare inside /etc/nixos/configuration.nix):

environment.systemPackages = [ pkgs.perl pkgs.emacs pkgs.thunderbird pkgs.firefox ];

Not much to say, packages will be downloaded from cache or compiled if binaries are not available.

Remember: when installing i.e. some perl application, in one environment, you will not see perl executable itself, you will have to specify it separately.

User environment

Example (as normal user, without sudo of course):

$ nix-env -iA pkgs.emacs24

This package will be installed to something like /nix/store/qv39sv3981kk5h4p260y879wilfc3c26-emacs-24.3 let’s see what is inside (two levels deep)

/nix/store/qv39sv3981kk5h4p260y879wilfc3c26-emacs-24.3
|-- bin/
|   |-- ctags*
|   |-- ebrowse*
|   |-- emacs -> emacs-24.3*
|   |-- emacs-24.3*
|   |-- emacsclient*
|   |-- etags*
|   `-- grep-changelog*
|-- libexec/
|   `-- emacs/
|-- share/
|   |-- applications/
|   |-- emacs/
|   |-- icons/
|   |-- info/
|   `-- man/
`-- var/
    `-- games/

You can see that everything from this package is in one place. Dependencies are also taken care of in similar manner and are reused for other packages of course.

Well how the hell do users see/use right package? The answer is simple: each user has its own profile which is a folder and links are created into it.

If some other user wants emacs23 and not emacs24, that is not the problem, dependencies are going to be reused and new links into other user profile will be created.

Custom environment

Now here the fun starts!

If you want emacs23 side by side with emacs24, you need to create a custom environment. In this example I will create two custom environments, one for each package, but you can have one emacs in user env. and one in custom env.

Edit/create a new file ~/.nixpkgs/config.nix:

{
  packageOverrides = pkgs:
  rec {
    homeEnv = pkgs.buildEnv {
      name = "homeEnv";
      paths = [ pkgs.emacs24 pkgs.bsdgames ];
    };
    workEnv = pkgs.buildEnv {
      name = "workEnv";
      paths = [ pkgs.emacs23 pkgs.perl ];
    };
  };
}

Then you need to build those two profiles:

$ nix-env -p /nix/var/nix/profiles/per-user/<YOUR_USERNAME>/workEnv -i workEnv

$ nix-env -p /nix/var/nix/profiles/per-user/<YOUR_USERNAME>/homeEnv -i homeEnv

I choose to create profiles inside /nix/var/nix/profiles/per-user/YOUR_USERNAME/ folder, but you can use ~/.profiles/ or something. The -i flag is the –install flag that we seen before and we are this time installing profiles named homeEnv and workEnv. Let us see what is inside homeEnv (this time one level deep):

/nix/store/rg4zh4s2gaw1b3dm2kgicis8admsaanx-homeEnv
|-- bin/
|-- libexec -> /nix/store/qv39sv3981kk5h4p260y879wilfc3c26-emacs-24.3/libexec/
|-- share/
`-- var -> /nix/store/qv39sv3981kk5h4p260y879wilfc3c26-emacs-24.3/var/

You can see that profile is actually created inside /nix/store and if some other user chooses to create the same packages for his profile, only one link will be created - his profile will be the same as yours.

One thing remains… create two more files, one for home environment

export PATH=/nix/var/nix/profiles/per-user/<YOUR_USERNAME>/homeEnv/bin
$@

and one for work environment

export PATH=/nix/var/nix/profiles/per-user/<YOUR_USERNAME>/workEnv/bin
$@

you will need to source those files to set environment variables for the profile or make them executable (also make them available in PATH env variable) and run commands as CLI arguments (hence the $@)

This was child’s play, environments are more useful in development, you can have one environment for Perl development and the other one for Python, of course in every profile, applications will see only each other, well that depends also on shell environment variables, for my Python 2.7 profile, this file now looks like this

#!/usr/bin/env bash

NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/matej
nixprofile=$NIX_USER_PROFILE_DIR/py27

export PATH="$nixprofile/bin"
export LD_LIBRARY_PATH="$nixprofile/lib"
export NIX_LDFLAGS="-L$nixprofile/lib -L$nixprofile/lib/pkgconfig"
export NIX_CFLAGS_COMPILE="-I$nixprofile/include -I$nixprofile/include/sasl"
export PKG_CONFIG_PATH="$nixprofile/lib/pkgconfig"
export PYTHONPATH="$nixprofile/lib/python2.7/site-packages"
export PS1="py27 $PS1"

export INCLUDE="$nixprofile/include:$INCLUDE"
export LIB="$nixprofile/lib:$LIB"
export C_INCLUDE_PATH="$nixprofile/include:$C_INCLUDE_PATH"
export LD_RUN_PATH="$nixprofile/lib:$LD_RUN_PATH"
export LIBRARY_PATH="$nixprofile/lib:$LIBRARY_PATH"
export CFLAGS=$NIX_CFLAGS_COMPILE
export LDFLAGS=$NIX_LDFLAGS

"$@"

This concludes this post and do please leave a comment bellow.

Jump into Hydra

Mon, 06 Jan 2014 00:00:00 +0100

Some basics to use Hydra through web interface.

If you have Hydra already up and running then you need to add some work to it.

You start on Hydra by creating a project, then you need a jobset, after that you include build inputs to it and Hydra will automatically start evaluation.

You confused now? Long version is here:

Project

Project is actually a group of Jobsets, and you can create it by clicking on Admin -> Create project, here you specify meta data and assign the owner of the project, the one who can configure jobsets within this project.

Jobset

Jobset contains build configuration (build inputs) and meta data. Create it by clicking Actions -> Create jobset, located on project page.

I think that some options are self-explanatory, the rest are here

Name	Description
State - Enabled	Hydra will check all build inputs periodically and start evaluation when it detects the change
State - One-shot	evaluations will not be automatically started you must start them by hand (located in jobset page: Actions -> Evaluate this jobset)
State - Disabled	disabled state, but you can start evaluation by hand (not sure if this is a bug).
Nix expression	this loads dummy_script.nix from build input named in this example: dummy_scripts
Scheduling shares	larger the number the faster evaluation of this jobset will start, considering you have more jobsets, with only one jobset this option has no effect
Build inputs	here we have one input named dummy_scripts which in this case is Git repository that includes file named dummy_script.nix, the rest of build inputs are available as parameters of this script

Job

Job is produced by Hydra, one jobset can contain multiple jobs (hence the name), they are created by scripts written in Nix expression language.

Now comes the example

This Nix expression script is useful when you wish to test some package(s) if their build process is successful and when you wish to share binaries with others.

Now make a new jobset, with this parameters…

Name	Value
State	Enabled
Visible	checked
Identifier	build_shells
Description	With this jobset we build shells.
Nix expression	build-general.nix in hydra_scripts
Check interval	30
Scheduling shares	10

and build inputs …

Name	Type	Value
hydra_scripts	Git checkout	git://gist.github.com/7328623.git
attrs	Nix expression	[ “bash” “zsh” ]
supportedSystems	Nix expression	[ “x86_64-linux” “i686-linux” ]
nixpkgs	Git checkout	git://github.com/matejc/nixpkgs.git

For nixpkgs build input you can use your own forked official NixOS/nixpkgs, I do not recommend using official repository for this because it is busy, and remember.. when I wrote that Hydra checks every build input for changes and then starts evaluating every jobset that has changed, well, in this case it will rebuild only if someone changes bash or zsh packages.

Now wait for the evaluation to pass, meanwhile take a look around the page.

When a build is successful you will see something like this

go to the Jobs tab and you should see something like this

click one Job, I choose ‘bash.i686-linux’, now click on the number preceding release name of the package to get to this page

On Summary tab the most interesting thing are Build products, where you can click on the Help button beside them and install that package on your computer.

This example is not very useful because bash and zsh are already available in NixOS cache, but with this script you can build any package in nixpkgs repository, including your own packages, even before you make an official pull request to the NixOS/nixpkgs repository and do not forget the Build products… with one command you can install that package on any computer that has Nix package manager installed.

NixOS + Hydra + Nginx

Mon, 23 Dec 2013 00:00:00 +0100

Setup Hydra - the build server - on NixOS with Nginx

NixOS is very special operating system which uses functional packagemanager called Nix, I bet that it does not mean much to you if you do not know how it works.. well here is a quick intro: to install NixOS you need to format the disk, make partitions, then write the configuration file for NixOS and run one command and system will build itself. When you want to change this system configuration, just edit the file and rerun the rebuild command anytime, it is very much like zc.buildout.

Hydra is a build farm for NixOS. Why NixOS needs a build farm, one reason is that every packet has its prefix like so

1	/nix/store/<hash>-<packetname>-<version>/

. That way you can have many versions of the same packet installed, without interference from each other, which is very useful for developers.

Well, Hydra can be used and abused for many things which I will write on this blog a bit later. Hydra can build any packet you can think of and on success it can run tests and on their success, build documentation, test coverage report, binary tarball, source tarball, with a bit of struggle you can build deb or rpm packets and off course all available on Hydra’s server which you can setup yourself on your server.

I have my Hydra here.

Here is the configuration.nix (edit as necessary):

{ config, pkgs, ... }:
let
  # make sure we always have the latest module
  hydra = pkgs.fetchgit {
    url = https://github.com/NixOS/hydra;
    rev = "refs/heads/master";
  };
in {

  ...

  require = [ "${hydra}/hydra-module.nix" ];

  ...


  services = {
    # for sending emails (optional)
    postfix = {
      enable = true;
      setSendmail = true;
    };

    # you are probably going to need openssh server
    openssh = {
      enable = true;
      permitRootLogin = "no";
      passwordAuthentication = false;
    };

    # actual Hydra config
    hydra = {
      enable = true;
      dbi = "dbi:Pg:dbname=hydra;host=localhost;user=hydra;";
      package = (import "${hydra}/release.nix" {}).build.x86_64-linux;
      hydraURL = "http://hydra.scriptores.com/";
      listenHost = "localhost";
      port = 3000;
      minimumDiskFree = 5;  # in GB
      minimumDiskFreeEvaluator = 2;
      notificationSender = "hydra@yourserver.com";
      logo = null;
      debugServer = false;
    };
    # Hydra requires postgresql to run
    postgresql.enable = true;
    postgresql.package = pkgs.postgresql;

    # frontend http/https server
    nginx.enable = true;
    nginx.config = pkgs.lib.readFile /root/nginx.conf;
  };
  ...
}

This is main configuration file for Nginx (stored in /root/nginx.conf)

#user  nobody;
worker_processes  1;

error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include /root/hydra.nginx;

}

We also need the included file hydra.nginx (stored in /root/hydra.nginx)

# ssl, mostly for people that are going to need to login to Hydra,
#      we do not want to send passwords as plain text
server {
    listen 0.0.0.0:443 ssl;
    server_name hydra-ssl.scriptores.com;
    keepalive_timeout    70;

    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  10m;
    ssl_certificate     /root/ssl/hydra.crt;
    ssl_certificate_key /root/ssl/hydra.key;

    ### We want full access to SSL via backend ###
    location / {
        proxy_pass http://127.0.0.1:3000/;

        ### force timeouts if one of backend is died ##
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;

        ### Set headers ####
        proxy_set_header        Accept-Encoding   "";
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        ### Most PHP, Python, Rails, Java App can use this header ###
        #proxy_set_header X-Forwarded-Proto https;##
        #This is better##
        proxy_set_header        X-Forwarded-Proto $scheme;
        add_header              Front-End-Https   on;

        ### By default we don't want to redirect it ####
        proxy_redirect     off;
    }
}

# redirect http to https
server {
    listen 0.0.0.0:80;
    server_name hydra-ssl.scriptores.com;
    rewrite ^ https://$server_name$request_uri? permanent;
}

# for normal folks
server {
    listen 0.0.0.0:80;
    server_name hydra.scriptores.com;

    location / {
        proxy_pass http://127.0.0.1:3000/;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Generate self signed certificate and save files to /root/ssl/ (make sure that .key file is not world readable and that CN is your actual domain!)

openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout hydra.key -out hydra.crt

Make sure you have /var/lib/hydra/.pgpass file, with this contents

localhost:*:*:hydra:yourHydraPgPassword

PostgreSQL configuration

$ sudo -u postgres createuser hydra -P
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

$ sudo -u postgres createdb -O hydra hydra

$ echo "GRANT ALL ON DATABASE hydra TO hydra;" | sudo -u postgres psql hydra

You need to init the database now.. You have to do this after each upgrade also, fortunately this is easy, run this:

hydra-init

Additional links:

Installing Hydra on Ubuntu

Also this is very usefull!

Matej Cotman's Blog - NixOS

Multi VPN setup

Goals

Linux Namespaces

What are Linux Namespaces?

Let’s see the solution

Usage

The Nix script (extended from nsjail.nix)

Parameters

Running

Conclusion

Neovim, WSL and Nix

Intro

Let’s see the solution

Neovim-Qt

WSL

Nix

The Nix script

Usage

Conclusion

Links

Graphical UI for Nix/NixOS

What is new?

Preview of 0.1.1 version

Front Page - Package Manager

Front Page - Configuration Options

Package Browser

Package Info

Manage Marked

Configuration Browser

How to try it out?

Development

Nixpkgs

Plans for the future

Too many Nix pills, I am addicted

Lets start with usage

Code

1 templatesFile

Generate package info

Lets go right to it!

Output

Conclusion

Nix's very own graphical UI

Well, let’s start…

Usage

Manage marked dialog

Plans for the future

Control your packages with Nix created profiles!

System environment

User environment

Custom environment

Jump into Hydra

Project

Jobset

Job

Now comes the example

NixOS + Hydra + Nginx

`1 templatesFile`