hpfriends - real-time social data-sharing

hpfriends - real-time social data-sharing

SIGINT 2013 presentation by Mark 'rep' Schloesser and Johannes 'heipei' Gilger


  1. About us
  2. Motivation
  3. hpfeeds
  4. hpfriends
  5. Technology

About us

Mark 'rep' Schloesser

  • Malware researcher at Rapid7
  • Honeynet member since 2008

Jojo 'heipei' Gilger

  • PhD student, IT-Security, RWTH Aachen University
  • Honeynet member since 2012


  • Lectures: ITSec 1+2, Mobile Security, Security in Wireless Networks
  • Lab: Security Lab (Hands-on practical offensive security)
  • CTF: 0ldeur0pe, rwthCTF 2011 & 2012


The Honeynet Project

The Honeynet Project is a leading international 501c3 non-profit security research organization, dedicated to investigating the latest attacks and developing open source security tools to improve Internet security.

Homepage: honeynet.org

Motivation: Sharing stuff

Hey Mark, can I have (real-time) access to the malware hitting your sensors?


Hey Jojo, can you give me access to the samples you get from Mark?

Other people

Sure, let me just ask the admin to give you access to the channel...

Jojo (probably not the last time)

Motivation: The HoneyMap

Available options for sharing data

The usual suspects...

... with their common problems.

Honeynet Generic Datafeeds

The hpfeeds protocol

hpfeeds is an authenticated publish/subscribe datafeed protocol (by Mark) used by the Honeynet Project. It supports channels and authkeys.

hpfeeds - details

hpfeeds - pros & cons



So what would we need?

A system for large-scale real-time data-sharing would have to ...

Distributed Access Control

Managing permissions for a large number of users does not scale and also does not correspond to the way we collaborate. Think "Web of Trust" and "Distributed Version Control" (like with git).

Social is the solution!


hpfriends is an evolution of the hpfeeds system. The wire-format is identical, the message broker behaves the same way. The big difference is the data-sharing model and the backend software.

hpfriends - unique features

hpfriends supports users and groups. These are connected by sharing relations, representing a web of trust between users who trust each other by sharing their own data as well as data received by third parties.

hpfriends - social graph

hpfriends operates on a social graph. Users and groups are nodes, sharing relations are edges.

Edges between users have a type (shares, shares_all) and an attribute, i.e. the channel name.

Group membership is also stored as an edge between a user and group.

hpfriends - channels

Channels behave quite differently than in the original hpfeeds.

hpfriends - demo time!

We're going to show you ...

  1. A simple 1:1 data sharing scenario
  2. A scenario where data is also relayed to a third party
  3. The way the HoneyMap works behind the scenes

hpfriends - implementation

Currently, hpfriends uses the Neo4j graph database to store user, group and sharing information. Further information on users / etc is stored within MongoDB.

Screenshot from Neo4j interface

For actual message delivery via the feedbroker, sharing relationships are cached for 5-minute intervals for performance.

hpfriends - Neo4j query

start usrc=node({srcid}), udst=node({subids})
        match p=usrc-[:*1..]->udst, udst-[ra:accepts]->umid
        with relationships(p) as rs, udst as udst, ra as ra
        where id(last(rs)) = ra.sid
        return distinct udst'''

hpfriends - frontend

Public hpfriends webif is at hpfriends.honeycloud.net. The webif was built using Meteor, a real-time JavaScript / NodeJS web-framework which takes care of data persistence & synchronization, data binding to the UI, templating, latency compensation and many other things.

hpfriends - hpfeeds wire format

struct MsgHeader {
  uint32_t messageLength; // total length
  uint8_t opCode; // currently five opCodes implemented
length | opcode | next | identifier | next | channelname | payload
    85        3   9      b4aa2@hp1    9      mwcapture     137941a3d8589f9

Full documentation and list of opCodes at github/rep/hpfeeds.

hpfriends - Software support

Your software as well?

hpfriends - links

Thank you for listening!

Follow-up questions to
Mark - @repmovsb
Jojo - @heipei