Oct 2005 to Mar 2006
1.0 DEPLOYMENTS
1.1 Current technologies deployed
During this reporting period we have continued to operate two active honeynets deployed in the UK. They remain a mixture of GenII (Eeyore) systems and GenIII (Roo) systems, all using the Honeynet Project’s bootable Honeywall CDROMs. Contrary to our previous plans, we decided to keep our older Eeyore honeynet running for an additional six months because we had an actively compromised group of GenII honeypot systems that we wished to continue monitoring. A range of honeypots were deployed within each of the two active UK honeynets during this reporting period.
Our deployment configurations were:
UKA – UK ISP ADSL (full class C network)
- Data Control and Capture
- Roo Honeywall
- Sebek v3
- Data Analysis
- Walleye web interface
- Honeywall daily analysis script + email alerting
- Honeysnap daily analysis script
- Honeypots
- Redhat 7.3 on Intel unpatched
- Fedora Core 1 on Intel unpatched
- Solaris 9 on Sun U10 unpatched
- Malware collection on Debian Intel (~250 IP) using either:
- Nepenthes v0.1.3 (~250 IP)
- Nepenthes v0.1.6 (~120 IP) and MWCollect v3.0.4 (~120 IP)
- Leurre.com UK node (4 IP)
During this reporting period we have routinely upgraded the UKA honeynet during Roo Honeywall development, testing many revised GenIII Roo Honeywall CDROM releases, up to and including the current version (and yum updates).
UKB – UK ISP Data Centre (partial class C network)
- Data Control and Capture
- Honeywall v0.69
- Sebek 2.1.7 or 2.5.3
- Data Analysis
- ACID console
- Sebek web interface
- Honeywall upload script
- Honeywall daily analysis script + email alerting
- Honeysnap daily analysis script
- Honeypots
- Redhat 7.3 unpatched
- Fedora Core 2 unpatched
- Solaris 9 on Sun U10 unpatched
- Solaris 8 on Sun U5 unpatched
UKB was finally taken down on 01/04/06 for an upgrade to Roo in the coming weeks, but depending on long term planning plus ISP circuit and hardware delivery lead times, it may possibly be replaced with “Honeynet Central” in the next reporting period.
1.2 Activity Timeline
TBC
2.0 Findings
2.1 Highlight any unique findings, attacks, tools, or methods
After observing very regular SSH mass scanning activity, and capturing various attack toolkits from a number of sources, in March we decided to modify one honeynet to focus on this activity. Accounts using some of the common username/password combinations were added to each honeypot. Subsequently mulitple modified honeypots were compromised by Romanian attackers and used to host PsyBNC IRC servers, run LAN sniffers and attempt to scan other local class A net blocks for other vulnerable hosts. Interestingly, only Intel Linux honeypots were compromised (not Sparc Solaris), and although the honeypots were hosted on a new UK IP address range, it appears that we had probably seen the attackers before in a previous (non-SSH brute force) compromise. The modified honeynet was taken offline after a few days for analysis, with a mini-report to follow (including analysis of some mass scanning tools).
We observed an attack against a fake PHPShell on a honeypot by Indonesian bot herders, who tried to install a bot locally and configure it via accessing a remote PHP controller page. This incident demonstrated that even very sparse, low interaction honeynet data can provide much detail about an attacker and will feature in future presentations/reports.
We have been running combinations of Nepenthes and MWCollect sensors for the past year, and as no direct performance and malware collection comparison statistics were available, we decided to conduct a research excercise on this topic. Two identical Debian Linux systems (both hardware and software) were set up, each running the default configuration of the latest version of Nepenthes and MWCollect. 120 IP addresses were assigned to each sensor, applied as virtual interfaces in an alternating fashion (.1 = Nepenthes, .2 = MWCollect, .3 = Nepenthes, etc) and connected to a 2MBit/sec UK ISP ADSL circuit. Malware collection data was gathered for a month and a short comparison report will be published shortly, although the importance of this data has been significantly reduced due to the recent Nepenthes / MWCollect Fusion announcement – timing is everything! 🙂
During March one Nepenthes sensor collected an unusual piece of Windows malware that was protected using PELock. At the time, other members of the Malware Collection Alliance had not seen this particular binary, or other exploit binaries protected using PELock, and the malware was not detected by common AV or sandbox tools such as Norman or VirusTotal. Initial malware analysis was performed by members of the French honeynet project and more detail will follow in a future mini-report.
We still regularly experience SQL Slammer activity on some UK networks, so after a brief local study we intend to run a more scientific investigation soon – ideally with the assistance of other Research Alliance groups too.
We have regularly analysed Rock Phish related phishing sites, and continue to maintain an active interest in the evolution of this phishing toolkit.
We have recently been working on Point of Sale (POS) systems and will soon be investigating their potential as high value honeypots.
2.2 Any trends seen in the past six months
Windows malware attack rates remained significant, with the expected mean time to compromise of an unpatched Windows XP host remaining between 5-10 minutes.
We continued to experience high volumes of SSH mass scanning activity, coupled with regular brute force or dictionary attacks, on many of our honeypot and production systems.
The end of 2005 saw a massive rise in the number of phishing scam related emails and other financial fraud attempts received. Phishing scams are investigated almost daily and we often get examples sent to us from other parties.We hope to continue to improve our data acquistion in the coming months, through more interaction with other international phishing tracking groups.
Compromised honeypots still almost always seem to have a PsyBNC IRC server installation, and the most common attacker language continues to be Romanian. Blackhat IRC traffic continues to suggest that mass scanning, compromising of a wide range of systems and financial fraud are common place on the Internet.
2.3 What are you using for data analysis? What is working well, and what is missing, what data analysis functionality would you like to see developed?
We had our first Linux honeypot with Sebek v3 and Walleye / Roo compromised last month, and the new capabilities assisted in speeding up the initial analysis (particularly the historical p0f data and associated flows, etc).
We still find Honeysnap to be the best initial daily indicator of when analysts need to spend time investigating activity on our honeypots. We released a final bugfix release of the shell script version recently, and development work is now being taken over by the Honeynet Project’s new Honeysnap dev team (of which the UK will hopefully continue to play a significant role).
Having strong, cutting-edge technology is great, but what you do with it is more important. We plan to go back over our historic data and see what extra value we can extract in the coming months (statistics, trends, KYE on IRC activity, etc). Previously we have focused on discrete incidents and events but we need to also complement this work at a macro level. Data analysis will be our core goal for the coming 6-12 months and we are currently planning our next steps. We have recently begun testing some small tactical extensions to Honeysnap (working title Honeymine) and hope to improve our capabilities in this area significantly during the next reporting period. Watch this space!
We would like an easy way to determine the source IP address relating to a particular keystroke (ie. Sebek data) from the command line – ideally a modification to sebek_extract + sbk_ks_log.pl that optionally inserted the source IP address as an field in each output line. Optionally the source OS version (via p0f) too would round things off nicely!
3.0 Lessons Learned
3.1 What new positive things can you share with the community, so they can replicate your success?
We stuggled to get Honeysnap working cleanly with Roo, but it turned out that the new Honeywall PCAP-API provides a very elegant was to extract arbitrary network packet capture data by date/time range. Kudos to Ed, Camilo and co for making our lives easier! 🙂
The malware collection tools continue to go from strength to strength, and with the MWCollect/Nepenthes Fusion announcement, it is very simple to easily collect the malware which is local to your own networks. Coupled with automated Norman Sandbox analysis reports, this represents an excellent way of keeping on top of malware activity without the need to deploy full high interaction honeypots.
We have found the TRAC/SVN system very useful for shared development and collaboration.
3.2 What new mistakes can you share with the community, so they don’t make the same mistakes?
Slightly embarrasingly, this one is much the same as the last report – when you “kill off” a honeypot/honeynet, don`t simply rely on the host remaining powered down and always remove the network cable too. That way, helpful ISP staff won`t accidentally re-activate a compromised system after a power outage!
If you are deploying mixed GenII/GenIII honeypots (such as when you still have Solaris Sparc honeypots running Sebek v2), remember that honeysnap expects a singular version of the sebek tools and might have problems with multiple sets of data from both versions at once.
Don`t run your nepenthes sensors with the default logging level unless you have a lot of disk space – it will quickly run out and you will lose binaries/shellcodes. If possible, actively monitor disk space on all honeynet components, and alert on shortages (as sudden large changes in free disk space is a sure sign that something unexpected has occured).
Roo could really do with a “Honeynet Project Approved” rpm repository for yum updates. This way all updates would be checked and signed off as safe for honeywall operation before being pushed out via yum, much like commercial IDS vendors using Debian/Snort, etc do. This would hopefully avoid any more nasty yum update surprises, and improve stability. Users can always take un-approved patches if they need to, but it helps to maintain a solid baseline of known good components.
3.3 Are there any research ideas you would like to see developed?
As per the comments above, we believe that improving our data analysis and attack characterisation/profiling capabilities is the key to increasing the breadth and depth of our understanding. We hope that the new Honeywall Data Analysis tools will help here, along with the Python port of Honeysnap, but we intend to drive research in this direction over the coming months.
Futher development and integration of web application honeypots such as Googlehack Honeypots or PHP.HoP would be ideal, as would development of web crawlers and other forms of Honeyclients.
We still want to see a Sebek v3 client for Sparc Solaris, since we may well have to stop deploying Sun honeypots if Sebek does not keep up with changes in Roo, and changes in data formats make duel v2/v3 sebek clients less desirable to deploy in the wild. Are there other people out there still operating Solaris Sparc Honeypots?
Finally, how about a MacOS version too, for the people out there interested in deploying Mac honeypots?
4.0 NEW TOOLS
4.1 What new tools or technology are you working on?
The new Python version of Honeysnap, to make the code more robust, scalable and supportable, plus extend the existing functionality.
Small, tactical tools (like Honeymine) to form the basis of our new automated data analysis and attack profiling approach. We intend to start building a local “forensic database” for tracking and correlating distributed data such as probes, attacks, userids, passwords, hostnames, URLs, downloaded files/shellcode, filesums/details, IRC channels, unique strings, etc. Once we have a working prototype we would like to look at making this more open and encouraging some form of (potentially sanitised) data sharing on a global level.
POS related honeynet technology.
Cacti / RRD into Roo for monitoring honeywall health and quickly viewing trends. We still plan to make more use of this in the coming months and will merge out current work into the upcoming Roo releases to add web based system management and capacity planning reporting and into Honeysnap for at-a-glance status monitoring.
We have just started experimenting with the concept of Honeynet Widgets – ie Mac or Windows desktop widgets for showing system status, alerts, etc. We would like to develop something in this area soon.
We will soon begin experimenting with honeynet alerts and data via Blackberry or SMS. We hope that this might improve our response to interesting honeynet activity.
4.2 Would you like to integrate this with any other tools, or you looking for help or collaboration with others in testing or developing the tool?
We`d like to see Honeysnap become a core part of the Honeywall, and eventually have the next generation analysis and profiling tools integrated too. Testing of Honeysnap on other non-UK live data streams would be useful.
Further testing of Honeystick, general comments on usage experience, etc would be appreciated – particularly with the recent changes in VMWare licensing around Player and Server.
We`d love to see the French Honeynet Project continue developing their VMWare VM obfuscation patches for the current code releases (hint, hint Laurent!)
🙂
5.0 PAPERS AND PRESENTATIONS
5.1 Are you working any papers to be published, such as KYE or academic papers?
Yes – KYE: Blackhat IRC (may become KYE: Romanians or KYE: Profiling, or another separate paper).
Also a number of other mini-reports to follow, as per comments above.
5.2 Are you looking for any data or people to help with your papers?
Yes – Good examples of blackhat IRC data for inclusion in the upcoming KYE paper.
Also people interested in improving our incident analysis and attacker profiling capabilities.
5.3 Where did you publish/present honeypot-related material?
- February 2006 – Credit Card Fraud Workshop, Chicago
- March 2006 – DoE/DoD/NSA Honeynet Workshop, Washington State
Upcoming:
- April 2006 – Honeynet Profiling Workshop, Washington DC
- April 2006 – Infosec Europe 2006, London
- TBC 2006 – York University
6.0 ORGANIZATIONAL
6.1 Changes in the structure of your organization
None.
6.2 Your feedback on Alliance activities.
Good to see lots of activity happening, particularly when face to face meetings occur. Also good to see a number of new international organisations coming onboard and contributing.
6.3 Any suggestions for improving the Alliance?
7.0 GOALS
This was an interesting period for the UK group, since due to unexpected day job work commitments (major project extensions, now complete) and temporary changes in honeynet activity focus, we ended up not starting many of the goals we set ourselves for this period. However, we did do a lot of other things during this period instead, we just didn`t do the things that we had planned at the expected time. In fact, almost all of our goals from last time remain our goals for this time, and we have now come full circle.
7.1 Which of your goals did you meet for the last six months?
- Design and begin deploying a much larger distributed honeynet system in the UK (“Honeynet Central”), if possible with funding
[met, but on hold to allow prioritisation of data analysis improvements] - Participate in DoD / DoE / NSA conference (Washington State Dec 2005)
[met – March 2006] - Make more time for honeynet research and other activity [met]
7.2 Which of your goals did you not meet for the last six months?
Basically everything from the previous report except the items above. One of the down sides of bi-annual reports but a flexible approach to monthly goals!
7.3 Goals for the next six months
Another cut and paste from our previous report – “Data analysis still remains our main concern. We are good at building and operating honeynets, but analysis of captured data is still very time consuming and requires too much human analyst time. Better data analysis and reporting tools, with greater cross group communication and incident response are essential. Near real time alerting and responses are also highly desirable, which again require more powerful automated analysis tools and techniques”. Although we have recorded fewer incidents during this period, this is still our biggest challenge today. Although interesting, the last six months seems have done little to improve the situation, but we hope that new tools and techniques will begin to make a difference.
Although we initially intended to roll out “Honeynet Central” immediately, in hindsight we have decided that if we still can`t fully analyse two sets of honeynet data in near real time, ten further honeynets collecting data are going to be of little use until we address this problem. Therefore, to avoid swamping ourselves with raw data, we have decided to focus initially on improving our data analysis and attacker profiling capabilities. Once we can do this on current and historical data sets, we`ll progress with expanding our honeynet deployments.
This section is a re-run of the last report, with some additional newer items too:External Systems
- Upgrade public website (Plone) and start to post material to it regularly / blog
- Use news publishing interface to keep news current and allow more people to contribute
- Upload statistics graphs / charts / data to the website automatically
- Add dynamic statistics and reporting
- Set up / revive a general mail list for interested parties
- Start regularly publishing basic attack statistics
Internal Systems
- Move internal Wiki to Plone
- Improve internal systems reporting
- Move to a more powerful dedicated server for data processing
- Improve data backups
Organisational
- Make more time for honeynet research and other activity
- Continue fundraising and seeking external sponsorship
- Bring in a couple of additional core team members
- Improve timeliness of incident response and data analysis
- Start “handlers diary” for incidents
- Improve contacts with CERTs, phishing data collectors, etc
- Develop contacts organisations such as Jill Dando Cybercrime Research Centre and National Hi-Tech Crime Unit
- Form connections with legal experts in the UK and assess legal position of honeynet data capture – particularly IRC
Events
- Honeynet Profiling Workshop in Washington DC (April 2006)
- Present at Infosec Europe 2006 (April 2006)
- IEEE S&P Honeynet Track at West Point (June 2006)
- Run at least one track at annual Honeynet meeting in Chicago (August 2006)
- Begin regular European honeynet meetings (starting in Paris this spring)
- Help to organise logistics for another UK Honeynet conference with the Network Defence team at GCHQ
Deployments
- Continue to operate existing Roo honeynets
- Upgrade UKB to GenIII (if Honeynet Central still on hold)
- Decide when to begin deploying a much larger distributed honeynet system in the UK (“Honeynet Central”)
- Future honeypots for UKA:
- Ground hog honeypot (self deploying virtual Windows XP honeypot that automatically extracts downloaded malware and restarts itself)
- Mac Mini honeypot (ready to buy hardware)
- XBox honeypot (ready to buy hardware)
- SGI honeypot (ready to deploy now)
- GSX Server honeypots with FreeBSD, WinXP, Win98 (ready to deploy now)
- Move to Nepenthes only for malware collection
RnD
- Continue developing Honeysnap features and working with Jed Haile to create stable production version for public release
- Move to python internally for as much development as possible
- Create new data processing scripts for log analysis and data presentation
- Begin developing “forensic database” concept and produce prototypes of Honeymine
- Deploy live HoneyPOS systems in the wild and analyse activity
- Release a number of useful tools and articles
- Complete KYE Blackhat IRC white paper
- Begin another KYE whitepaper (Profiling?)
- Continue developing improvements to HoneyStick
- Investigate automatic deployment / decommissioning of shortlived honeypots
- Evaluate building a dedicated IRC analysis tool
8.0 MISC ACTIVITIES
8.1 Anything else not covered you would like to share.
In March we deployed our first Leurre.com node in the UK. Greetings to Marc and his team, we are looking forward to seeing what interesting data we can contribute.
Thanks to Pedro and the Portuguese Honeynet Project for their HoneyMole tool. Definitely a step in the right direction for honeyfarm delployment and distributed operation. We hope to use it in production soon.
Work has continued on the Honeynet Project internal server over the last six months, mainly around keeping Plone up-to-date, general systems updates and administration, creation and administration of mailing lists in Mailman, and maintaining backups. The Plone site has been extended with modules to allow polling / voting, and we hope these will prove useful in encouraging feedback. Further work is planned; the main aim is to resolve the problems preventing us from running a Plone-based repository for large files (eg. VMware images) and from there to look at other methods with which we might enhance communication.