Building Nepenthes on the OpenWRT embedded platform

Building Nepenthes on the OpenWrt embedded platform

When wanting to build Nepenthes on the OpenWrt platform for embedded devices I hit a few issues long the way and it took me some time to work out the best method for reliably setting up a build environment, compiling any software dependencies and testing updated flash images. Like many things, once you work out the correct process it all seems pretty simple, but to save other people wasting their time on trial and error testing, he is a quick HOWTO for building Nepenthes on the OpenWrt platform.

What is OpenWrt?

OpenWrt is described by it’s authors as a Linux distribution for embedded devices. Instead of trying to create a single, static firmware, OpenWrt provides a fully writable filesystem with package management. This frees developers from the application selection and configuration provided by the vendor and allows them to customize the device through the use of packages to suit any application. OpenWrt is a framework for building an application without having to build complete firmware around it, allowing consumer devices to be quickly repurposed for other un-intended uses.

http://openwrt.org

Why might I want to use OpenWrt with Nepenthes?

Whilst you can install some common Unix distributions (such as Debian) on low power embedded devices, they have been build for desktop or server usage and typically come with relatively large deployment footprints. Nepenthes and other low-interaction honeypot software will install on such systems, but given the limited storage and processing power available on embedded devices, a dedicated embedded OS makes more sense. For example, the entire Nepenthes package for OpenWrt is only 2.1 Mbytes, which is much smaller than individual include libraries under traditional Debian. Despite being an embedded device OS, OpenWrt comes with basic Unix functionality and shouldn’t be a major problem for anyone with normal development and systems admin experience.

Do you recommend any particular hardware?

The current supported set of OpenWrt hardware can be found here:

http://wiki.openwrt.org/TableOfHardware

For running low interaction honeypot software locally on the embedded device sensor, I was looking for the best trade off between power and cost. At the time of writing (May 2008) this appeared to be the Linksys NSLU2 device, which provided a 266MHz Intel XScale CPU with 8 MBytes of flash and 32MBytes of RAM plus support for additional storage via USB devices (as opposed to only 4 MBytes of flash and 16 Mbytes of RAM and a 200MHz Broadcom chipset on the previously popular Linksys WRT54GL, which also works with Nepenthes).

If OpenVPN acceleration is required, alternatives such as the VIA C7 based systems with Padlock hardware AES encryption acceleration might be more suitable.

How do I set up my OpenWrt build environment?

To ensure that the OpenWrt build environment was clean and portable, I used a new VMWare virtual machine with the default Linux configuration except 20GB of hard disk space (may not be necessary and 8GB default may work too).

Steps:

Boot from latest Debian ISO (version 4.0r3)
Select standard system options
Add user openwrt
Once complete, reboot and log in as openwrt
Install initial software dependencies

aptitude update
aptitude upgrade
aptitude install openssh-server subversion gcc g++ libncurses5-dev binutils patch bzip2 flex bison make gettext pkg-config unzip zlib1g zlib1g-dev libc6 libc6-dev gawk autoconf upslug

Check out current OpenWrt release from svn

svn co https://svn.openwrt.org/openwrt/tags/kamikaze_7.09

Set up build environment (including non-open source Intel NPE Library)

cd kamikaze_7.09/
make defconfig
make package/symlinks
mkdir dl
cd dl
wget http://downloadmirror.intel.com/12954/eng/IPL_ixp400NpeLibrary-2_4.zip

Set up Nepenthes dependencies (ADNS). Beware of tab/space conversion issues when cutting and pasting from a web page!

mkdir package/adns
mkdir package/adns/patches
unexpand --first-only - >package/adns/Makefile
<cut and paste the following>

---begin---

#
# Copyright (C) 2008 David Watson
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.

include $(TOPDIR)/rules.mk

PKG_NAME:=adns
PKG_VERSION:=1.4
PKG_RELEASE:=1

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=ftp://ftp.chiark.greenend.org.uk/users/ian/adns/
PKG_MD5SUM:=88bc7bbf3f62a8d4fb186b8f72ead853
PKG_CAT:=zcat

PKG_BUILD_DIR:=$(BUILD_DIR)/$(PKG_NAME)-$(PKG_VERSION)
PKG_INSTALL_DIR:=$(PKG_BUILD_DIR)/ipkg-install

include $(INCLUDE_DIR)/package.mk

define Package/adns/Default
  SECTION:=net
  CATEGORY:=Network
  URL:=http://www.chiark.greenend.org.uk/~ian/adns/
endef

define Package/adns
  $(call Package/adns/Default)
  DEPENDS:=+libadns
  TITLE:=cli tools to resolve domains asynchronous
endef

define Package/libadns
  $(call Package/adns/Default)
  SECTION:=libs
  CATEGORY:=Libraries
  DEPENDS:=+libopenssl +zlib
  TITLE:=A asynchronous domain name resolver library
endef

define Build/Configure
        $(call Build/Configure/Default, \
                --prefix=$(PKG_INSTALL_DIR) \
        );
endef

define Build/Compile
        $(MAKE) -C $(PKG_BUILD_DIR) \
                DESTDIR="$(PKG_INSTALL_DIR)"
endef

define Build/InstallDev
        $(CP) $(PKG_BUILD_DIR)/src/adns.h $(STAGING_DIR)/usr/include/
        mkdir -p $(STAGING_DIR)/usr/lib
        $(CP) $(PKG_BUILD_DIR)/dynamic/libadns.so* $(STAGING_DIR)/usr/lib/
endef

define Build/UninstallDev
       mkdir -p $(STAGING_DIR)/usr/lib
       mkdir -p $(STAGING_DIR)/usr/include
       rm -rf  $(STAGING_DIR)/usr/include/adns.h
               $(STAGING_DIR)/usr/lib/libadns.{a,so*}
endef

define Package/adns/install
       install -d -m0755 $(1)/usr/bin
       $(CP) $(PKG_INSTALL_DIR)/usr/bin/adns $(1)/usr/bin/
endef

define Package/libadns/install
       install -d -m0755 $(1)/usr/lib
       $(CP) $(PKG_INSTALL_DIR)/usr/lib/libadns.so.* $(1)/usr/lib/
       install -d -m0755 $(1)/usr/lib
       $(CP) $(PKG_BUILD_DIR)/dynamic/libadns.so* $(1)/usr/lib/
endef

$(eval $(call BuildPackage,adns))
$(eval $(call BuildPackage,libadns))

---end---
<Hit enter then Control^D to save the file>

Set up Nepenthes source (this will install Nepenthes in /opt/nepenthes). Beware of tab/space conversion issues when cutting and pasting from a web page!

mkdir package/nepenthes
mkdir package/nepenthes/patches
unexpand --first-only - >package/nepenthes/Makefile
<cut and paste the following>

---begin---

#
# Copyright (C) 2008 David Watson
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.

include $(TOPDIR)/rules.mk

PKG_NAME:=nepenthes
PKG_VERSION:=0.2.2
PKG_RELEASE:=1

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=@SF/nepenthes
PKG_MD5SUM:=unknown

include $(INCLUDE_DIR)/package.mk

define Package/nepenthes
  SECTION:=net
  CATEGORY:=Network
  DEPENDS:=+libcurl +libmagic +libpcre +libadns +uclibcxx
  TITLE:=Nepenthes
  URL:=http://nepenthes.sourceforge.net
endef

define Package/nepenthes/description
        Nepenthes
endef

CONFIGURE_ARGS += \
        --with-libpcre="$(STAGING_DIR)/usr" \
        --with-libmagic="$(STAGING_DIR)/usr" \
        --with-libcurl="$(STAGING_DIR)/usr" \
        --with-libadns="$(STAGING_DIR)/usr" \

CONFIGURE_VARS += \
        CPPFLAGS="$$$$CPPFLAGS -I$(STAGING_DIR)/usr/include/uClibc++" \
        CXXFLAGS="$$$$CXXFLAGS -fno-builtin -fno-rtti -nostdinc++" \
        LIBS="-nodefaultlibs -luClibc++" \

define Build/Configure
        $(call Build/Configure/Default,)
        $(SED) 's|^sys_lib_dlsearch_path_spec=.*|sys_lib_dlsearch_path_spec="$(STAGING_DIR)/usr/lib"|g' \
                $(PKG_BUILD_DIR)/libtool
        $(SED) 's|^postdeps=.*|postdeps=|g' $(PKG_BUILD_DIR)/libtool
endef

define Build/Compile
        $(MAKE) -C $(PKG_BUILD_DIR) DESTDIR="$(PKG_INSTALL_DIR)" all
        $(call libtool_fixup_libdir,$(PKG_INSTALL_DIR))
        $(MAKE) -C $(PKG_BUILD_DIR)  DESTDIR="$(PKG_INSTALL_DIR)" install
endef

define Package/nepenthes/install
        install -d -m0755 $(1)/opt/
        install -d -m0755 $(1)/opt/nepenthes
        install -d -m0755 $(1)/opt/nepenthes/bin
        install -m0755 $(PKG_INSTALL_DIR)/usr/bin/nepenthes $(1)/opt/nepenthes/bin
        install -d -m0755 $(1)/opt/nepenthes/etc/
        install -d -m0755 $(1)/opt/nepenthes/etc/nepenthes/
        cp -R $(PKG_INSTALL_DIR)/etc/nepenthes/* $(1)/opt/nepenthes/etc/nepenthes
        install -d -m0755 $(1)/opt/nepenthes/lib/
        install -d -m0755 $(1)/opt/nepenthes/lib/nepenthes/
        cp -R $(PKG_INSTALL_DIR)/usr/lib/nepenthes/*.so $(1)/opt/nepenthes/lib/nepenthes
        install -d -m0755 $(1)/opt/nepenthes/var/
        install -d -m0755 $(1)/opt/nepenthes/var/cache/
        install -d -m0755 $(1)/opt/nepenthes/var/cache/nepenthes/
        install -d -m0755 $(1)/opt/nepenthes/var/cache/nepenthes/signatures/
        cp -R $(PKG_INSTALL_DIR)/var/cache/nepenthes/signatures/shellcode-signatures.sc $(1)/opt/nepenthes/var/cache/nepenthes/signatures/shellcode-signatures.sc
        install -d -m0755 $(1)/opt/nepenthes/var/binaries/
        install -d -m0755 $(1)/opt/nepenthes/var/log/
endef

$(eval $(call BuildPackage,nepenthes))

---end---
<Hit enter then Control^D to save the file>

Modify the Nepenthes source to remove Werror statements (which otherwise cause uClibc++ to abort on compilation warnings). Note that this could also be done in the Nepenthes Makefile using a nested $(SED) to remove Werror statements from all Nepenthes source sub-Makefiles.

cd dl
wget http://downloads.sourceforge.net/nepenthes/nepenthes-0.2.2.tar.gz
tar xvfz nepenthes-0.2.2.tar.gz
mv nepenthes-0.2.2.tar.gz nepenthes-0.2.2.tar.gz.orig
cd nepenthes-0.2.2/
for i in `find . | grep Makefile`
do
  sed -e 's/ -Werror//g' $i > /tmp/.nep
  mv /tmp/.nep $i
done
cd ..
tar cvfz nepenthes-0.2.2.tar.gz ./nepenthes-0.2.2
cd ..

How do I compile Nepenthes on OpenWrt?

Build your new packages and flash image using the build environment you set up previously (make sure you use the option “*” and not “M” to select a module, to ensure the package is included now and not left for later). Compiling the initial packages should take 30-60 minutes initially, depending upon your CPU speed, but you only have to do this once

make menuconfig
        Target = Intel XScale IXP4xx [2.6]
        Target Profile = Linksys NSLU2
                Base System
                        No bridge
                        No dnsmasq
                        No iptables
                Network
                        nepenthes
                        tcpdump
                        No ppp
                Libraries (should already have been automatically added)
                        libadns
                        libcurl
                        libmagic
                        libpcap
                        libpcre
                        uclibcxx
                Kernel
                        No wireless drivers
        Select "Yes" to save changes
make

What is the fastest/best method of testing new OpenWrt flash images?

Put your NSLU2 device into upgrade mode (power off, hold down reset button, power on whilst continuing to hold reset button for 10 seconds until power light turns red/deeper orange)
Install the new flash image onto your NSLU2 using the upslug tool

upslug2 -f <flash_image_file_name, such as openwrt-nslu2-2.6-squashfs.bin>

However, an even easier method (especially if you want to avoid bricking a working device with suspect firmware) is to compile your code for the x86 platform and then boot the resulting ext2 image in VMWare (http://wiki.openwrt.org/RunningKamikazeOnVMwareHowTo)

How do I get Nepenthes to start automatically on reboot?

You need to set up an init script and link it:

vi /etc/init.d/nepenthes

---begin---

#!/bin/sh /etc/rc.common
 # Copyright (C) David Watson
 START=50
start() {
 /opt/nepenthes/bin/nepenthes -c /opt/nepenthes/etc/nepenthes/nepenthes.
 conf -v -w /opt/nepenthes -D
 }
stop() {
 killall nepenthes
 }

---end---

chmod 755 /etc/init.d/nepenthes
ln -s /etc/init.d/nepenthes /etc/rc.d/S50nepenthes

Building embedded Nepenthes sounds complicated – is there an easier solution?

You can download pre-prepared build files for Nepenthes v0.2.2 here.

You can download a pre-compiled NSLU2 flash image for Nepenthes v0.2.2, which was built as per the process described above, here.

PLEASE NOTE THAT THESE UNSUPPORTED FILES ARE PROVIDED PURELY TO HELP OTHERS AND NO GUARANTEES ARE PROVIDED ABOUT THEIR FUNCTIONALITY OR RELIABILITY. USE AT YOUR OWN RISK – WE ARE NOT RESPONSIBLE IF YOU BRICK YOUR EMBEDDED DEVICE!

Failing that, try Debian on the NSLU2. Nepenthes v0.1.7 is available within the package repository and v0.2.2 also builds cleanly if you follow the instructions carefully:

http://www.nslu2-linux.org/wiki/Debian/HomePage

What do these common errors actually mean, and how do I resolve them?

1) “C Compiler can’t create executables” when making a new package.

Your code is c++ code, but your OpenWrt build environment only has a regular c compiler support enabled. To build Nepenthes for OpenWrt you need to ensure that the uClibc++ library is also used by the compiler, by specifying it explicitly when you set up your package options. Nmap is a good test tool to add, since the package Makefile is provided and it has similar dependencies as Nepenthes. If you can make that, you should be able to make Nepenthes too.

2) “File not found, SHUTTING DOWN” when trying to start Nepenthes on OpenWrt.

[ info mgr ] Loaded Nepenthes Configuration from "/etc/nepenthes/nepenthes.conf".
[ crit mgr module ] Failed to load library "lib/nepenthes/dnsresolveadns.so": File not found
[ crit mgr module ] ERROR LOADING MODULE lib/nepenthes/dnsresolveadns.so: SHUTTING DOWN
Quit
run is done -1

The “File not found” error is actually uClibc++ being rather confusing. What it actually means is that uClibc++ is unable to correctly resolve all linking dependencies that Nepenthes depends upon. To fix it you need all correct paths and correctly compiled shared objects / modules (ie with uClibC++ library support, rather than stdc++). Usually the cause of this problem is that not all your required libraries are linked with uClibc++ rather than stdc++ (which is the default for OpenWrt). Libtool seems to be the culprit, ignoring some compiler build flags and continuing to set stdc++ in the postdep configuration options:

cd /home/openwrt/kamikaze_7.09/build_armeb/nepenthes-0.2.2
grep postdeps libtool
        postdeps="-lstdc++ -lm -lgcc_s -lc -lgcc_s"

The Nepenthes Dev Team had the same problem when they looked at the same process, and Markus Koetter kindly helped work out the fix (which is to set up your Nepenthes Makefile correctly to ensure uClibc++ is used instead of stdc++ by modifying the libtool postdep statements via $(SED) statements). Thanks too Markus for this and other help! 🙂

3) Unable to load configuration file on start-up.

Unable to load configuration file "/etc/nepenthes/nepenthes.conf": No such file or directory
Quit
run is done -1

You need to correctly tell Nepenthes where to find it’s configuration files and set it’s working directory:

/opt/nepenthes/bin/nepenthes -c /opt/nepenthes/etc/nepenthes/nepenthes.conf -v -w /opt/nepenthes

4) Syntax “error near unexpected token” on initial build

make
  make[2] -C /v/build/bgm/OpenWrt-SDK-brcm-2.4-for-Linux-i686 package/compile
  make[3] -C package compile
  make[4] -C package/ruby compile
  make[4] -C package/ntpclient compile
/bin/sh: -c: line 4: syntax error near unexpected token `('
/bin/sh: -c: line 4: `          xargs -n1 cmd() { &>/dev/null make $* || {  echo "Build failed. Please re-run make with V=99 to see what's going on"; false; } } 3>&1 4>&2; cmd compile -C; \'

See patch 2508:

https://dev.openwrt.org/ticket/2508

5) POSIX shell build failures:

$ make -n
cmd() { &>/dev/null make $* || {  echo "Build failed. Please re-run make with V=99 to see what's going on"; false; } } 3>&1 4>&2; cmd package/compile
/bin/sh: Syntax error: "&" unexpected
make: *** [world] Error 2

See patch 2659:

https://dev.openwrt.org/ticket/2659

Do I have to recompile everything each time, or can I easily just recompile the individual package that I’m working on?

To make a single package (faster):

cd kamikaze_7.09 ; make TOPDIR=$(pwd) -C package/nepenthes V=99

Are there any known issues with this build process?

Yes. 🙂

1) Missing Separators on compilation

If you cut and pasted code from this web page and didn’t follow the advice on avoiding tab/space conversion problems, you will get the following error on compilation:

'missing separator:79'

This can be resolved by correctly copying the pre-formatted code (or using the downloadable examples in the tar file).

2) All downloaded malware samples have file type “null” and are not submitted to file

There appears to be a problem with libmagic causing all downloaded malware samples to have a file type of null and to therefore not be submitted to disk by default. This can be resolved by setting:

        strictfiletype              "0";

in nepenthes.conf and restarting Nepenthes.

3) Segmentation faults when collecting SQLSlammer samples

The Nepenthes SQLSlammer vulnerability module segfaults on the NSLU2 each time a sample is collected, killing Nepenthes. It appears that this is related to network byte ordering and endian-ness, and the Nepenthes dev team are aware of it. You can avoid this by commenting out the SQLSlammer module in /opt/nepenthes/etc/nepenthes/nepenthes.conf:

//    "vulnmssql.so",                 "vuln-mssql.conf",              ""

Alternative Embedded Platforms – OpenWrt on x86 and VMWare

If you are running multiple low and high interaction honeypots on a single host and memory availability is becoming an issue, one potentially interesting solution for reduced memory footprint malware collection is to deploy an OpenWrt image build specifically for the x86 embedded environment. A suitable ext2-based filesystem image can be constructed by following exactly the same steps as the process outlined above but setting x86 as the target architecture:

make menuconfig
        Target = Intel x86 [2.6]
        Target Profile = Default

You can then use QEMU’s qemu-img tool to convert the resulting ext2 file system image into a ready to boot VMWare .vmdk format disk image. A 32Mbyte RAM OpenWrt based Nepenthes collector will be substantially smaller in deployment footprint that a comparative Debian/Redhat based full OS installation. The main trade off is in the lack of simple upgradability and increased complexity of remotely deploying new disk images.

For further details on running OpenWrt on VMWare, see:

http://wiki.openwrt.org/RunningKamikazeOnVMwareHow

Alternative Embedded Platforms – OpenWrt on the Mikrotic Routerboard 532a

I’ve been testing a number of embedded platforms, and the Mikrotic Routerboard 532a seemed to offer a good balance of increased RAM, bootable NAND, multiple NICs and support for booting CompactFlash for little extra cost. An installation guide can be found here:

http://wiki.openwrt.org/OpenWrtDocs/Hardware/Mikrotik/RB532

Although the stock Kamikaze images from the openwrt web site didn’t work for me, these did:

http://wifi.ozo.com/airo/openwrt/firmware/kamikaze/2.6/rb5xx/stable/

Netbooting the device was relatively simple, requiring only the addition of a DHCP and TFTP server on Debian and some basic DHCP netboot configuration:

host 532a {
        fixed-address 192.168.0.154;
        hardware ethernet 00:0C:42:0E:9F:F3;
        next-server 192.168.0.10;
        filename "/srv/tftp/openwrt-rb532-vmlinux";
}

With the downloaded OpenWrt image file placed in /srv/tftp the device should netboot automatically when prompted via the serial console.

Once again, cross compiling our own custom Nepenthes image was as simple as changing the OpenWrt build target to the required platform:

make menuconfig
        Target = Mikrotik RouterBoard 532 [2.6]
        Target Profile = Default

Performance and reliability of my 64M/128M 532a seems good, although the two stage process flash the device (netboot then manually mount and extract file system/kernel image) is less elegant than simply running upslug2 for the NSLU2

Both examples above demonstrate one of OpenWrt’s main benefits: once you have a working build root, cross compiling for any of the 10+ supported embedded architectures is as simple as changing a couple of parameters in the make file and recompiling. Definitely easier than natively compiling for each target environment!