Netshare: Mount NFS v3/4, EFS or CIFS/Samba volumes in Docker

Recently I had a project where I was dealing with large file systems that had to be spanned and preserved against 100’s of Docker services for computation. I decided to use Amazons EFS which was recently announced at re:Invent 2015.

I setup the file system which was easy and attempted to play with it. I first mounted it manually via NFS4 on one of the Mesos
slaves. It worked and initial tests showed great performance. Now
the next step was meeting some of my personal requirements before I could call this successful.

My requirements were:

The file system should be mounted when needed within a Docker container and tore down when no longer in use
EFS provides endpoints in each availability zone so I also needed to support dynamic linking based on which slave the container was running in and the corresponding endpoint matching the AZ.
Must use NFS4 (This is what EFS supports)
Did not want to create or modify existing container images to attempt to mount NFS within the container itself. This also would mean the container would need to run in privileged mode which I’m against

After analysis, I decided the best choice was to write a plugin myself to extend the Docker engine.

Docker Volume Plugins

Docker volume plugins enable Docker deployments to be integrated with external storage systems. A volume plugin makes use of the -v and --volume-driver flag on the docker run command. For example:

$ docker run -ti -v volumename:/data --volume-driver=some_plugin ubuntu sh

This command passes the volumename through to the volume plugin as a user-given name for the volume.

By having the user specify a volumename, a plugin can associate the volume to an external volume beyond the lifetime of a single container or container host. This can be used, for example, to move a stageful container from one server to another.

Meet Docker-Volume-Netshare Plugin

Before I began coding a plugin I wanted to pre-define some requirements. Here’s some of them below:

Write it in Go so it runs at native speed
Support NFS 3/4 and CIFS (might come in handy someday)
For EFS support the plugin will lookup local EC2 metadata on the slave to find the corresponding AZ and region for building the URI from the File System ID
Must support volume management
- Understand when a volume is already mounted and keep track of its connections
- If a container is stopped determining based on active connections if the volume mount should be removed
Support all of the Docker Plugin API

After designing the plugin I installed it on the slaves (container hosts) and as a result I was able to dynamically mount EFS volumes and link them into the container based on the specified target mount.

Using Netshare:

First thing to do is download the binary for your particular platform at: https://github.com/ContainX/docker-volume-netshare

The examples below are just a few and the plugin offers many options. Let’s start with EFS.

Mounting EFS Volumes:

The example below will start a container and mount the specified EFS File System by the FS_ID (assigned by AWS) into a directory called /data in the container.

Run the plugin or add it to systemd
```
$ sudo docker-volume-netshare efs
```

Run a container to test:

$ docker run -i -t --volume-driver=efs -v aws_fs_id:/data ubuntu /bin/bash

Mounting CIFS/Samba Volumes:

The example below will start a container and mount the specified CIFS server/sharename into a directory called /data in the container.

Run the plugin or add it to systemd

$ sudo docker-volume-netshare cifs --username user --password pass --domain domain

Run a container to test:

$ docker run -i -t --volume-driver=cifs -v hostname/share:/data ubuntu /bin/bash

Mounting NFS Volumes:

Run the plugin or add it to systemd
```
$ sudo docker-volume-netshare nfs
```

Run a container to test:

$ docker run -i -t --volume-driver=nfs -v hostname/volume:/data ubuntu /bin/bash

Whats Next…

Here are some future enhancements I will be adding to the plugin based on requests and priority:

Support for credential files based on hosts or possibly a credential service
REST service to show stats on mounted volumes and how many connections
RPM packages for enterprise linux
Systemd and init based scripts

For more information see full documentation at: https://github.com/ContainX/docker-volume-netshare

Chris Schafer

So how would you feel about hosting a DB on this mounted storage? Specifically CIFS but possibly any of them? I have looked at MariaDB support remote storage of table files. Do you think using your tool and a CIFS/NFS NAS would be a reasonable and safe solution for production? No data on the CIFS share would have multiple access just getting the disk off the local host where it can be snapped and backed up. Great looking tool hoping it will meet my needs.
- Jeremy Unruh
  
  So the biggest problem with CIFS in the past has been with locking and stability. Personally I wouldn’t use CIFS but I have used NFS with great success. This is just my personal observations and I’m no expert with remote filesystems. If you have a fast network I/O and low latency then this can work great. I’ve mounted NFS against MongoDB and Redis for clustering and its worked great. I would experiment in your own environment and test the reads/write speeds to see if the numbers will scale for your normal usage. If you play around and see additional features that could be added please feel free to file an issue on GitHub.
abhishek kumar

Actually I want to mount a dir from within the docker with version 3, Where can I specify the NFS version? By default it is taking version 4.
- Jeremy Unruh
  
  You can launch it by going: ./docker-volume-netshare nfs -v 3. For full help on each command you can run it with ./docker-volume-netshare nfs -h for command flags/info
  - abhishek kumar
    
    Thanks for the help Jeremy.
    Even after launching with the “docker-volume-nfsahare nfs -v 3”, I am getting this error,
    
    INFO[2188] Mounting NFS volume 10.xx.xx.xx:/logs on /var/lib/docker-volumes/netshare/nfs/10.xx.xx.xx/logs
    
    2016/02/08 16:28:42 mount.nfs4: Protocol not supported
    
    INFO[2188] Unmounting volume 10.xx.xx.xx:/logs from /var/lib/docker-volumes/netshare/nfs/10.xx.xx.xx/logs
    
    2016/02/08 16:28:42 umount: /var/lib/docker-volumes/netshare/nfs/10.xx.xx.xx/logs: not mounted
    - Jeremy Unruh
      
      Try adding the –verbose flag so we can see what command is actually being executed. Also maybe there is a bug in the “-v” flag. Try it’s alternate of –version=3 . I don’t have access to my test environment right this minute but will look into it tomorrow with the extra details.
      - abhishek kumar
        
        It worked, actually the problem was, I had not stopped the previous daemon. Now after starting it is working perfectly fine. Thanks a lot 🙂
abhishek kumar

How can I provide the nfs mount options wsize, rsize manually? And what size you are using.
Ulf Michaelis

I try to mount a nfs-share using docker-compose.
The share itself works fine with the netshare-plugin and docker on the command-line. But I have no succes configuring docker-compose the rigth way.
currently i use a named volume in the service configuration:
volumes:
– logdata:/mounts/external/logs
and a seperate volume-section below:
volumes:
logdata:
driver: nfs
driver_opts:
basedir: “nfshostl/path/to/share”
The only message I got so far is “ERROR: VolumeDriver.Mount: exit status 32”

docker: 1.10.3
docker-compose: 1.6.2
docker-volume-netshare: no version avaliable, got it these days directly form the github-site.

Can you give me a clue?

Thanks in advance,
Ulf

Docker Volume Plugins

Meet Docker-Volume-Netshare Plugin

Using Netshare:

Whats Next…

Related