Smart Collection

The web application Huginn [1] continuously monitors several user-defined websites and Internet services. On the basis of the information published there, it draws up a summary, which it then mails to your breakfast table. In its report, Huginn might refer to the weather for your location or send an alert when Twitter sees many posts on a keyword such as net neutrality. You might receive an abstract like, "Twitter yesterday had an unusually high number of posts on net neutrality, the new XKCD comic published last night is all about drones, and the weather will be rainy today."

Much like the Internet service IFTTT [2], Huginn automatically triggers actions for certain events (e.g., if the text on a news page changes or the online shop reduces the price of the laptop you have your sights on). It is also useful for Internet of Things (IoT) projects. With a few exceptions, users access Huginn through a web interface.

Sophisticated

Huginn is available under the MIT license and runs on your server. It needs at least 2GB of RAM and a dual-core processor. If you want to run Huginn on a low-power computer like the Raspberry Pi, you need to manage the software's zest for action through configuration. The required modifications are explained in the Huginn wiki [3].

The software is based on Ruby on Rails [4] and therefore requires an installed Ruby environment; in fact, it needs version 2.2 or 2.3 of the reference implementation. The alternative Ruby implementations JRuby and Rubinius will not do the job.

Huginn's acquired data ends up in a MySQL or PostgreSQL database. The web interface delivers Nginx [5], but it should also work with an Apache server without any problems [6]. The developers only guarantee smooth operation on Debian 7 or 8 and Ubuntu 14.04 or 16.04. In principle, the installation also works on other Linux distributions, as well as OS X and FreeBSD.

Installing Huginn requires some work (see also the "Fast, Faster, Docker" box), and Ubuntu 16.04 has a nasty stumbling block: Huginn looks for runit , which resides in the repositories, but requires Upstart as the init system. However, Canonical replaced Upstart with systemd, causing Ubuntu 16.04 to report an error when installing runit .

docker run -it -p 3000:3000 huginn/huginn

docker run -it -p 3000:3000 -v /home/huginn/mysql-data:/var/lib/mysql huginn/huginn

If you still want to use Ubuntu 16.04, build runit yourself or ignore the error message. In our lab, however, runit ran flawlessly. Huginn also fails on the current Ubuntu 17.04, where the Foreman tool crashes reproducibly when creating the init scripts. In the end, Huginn only ran smoothly on Ubuntu 14.04 and Debian 8 (Jessie).

Package Services

In Debian, you first log in as root and run the sudo command-line tool, which lets you run commands under a different user account:

su root
apt install sudo

You will need a text editor, such as Nano, used in this example. The commands from Listing 1 retroactively resolve all dependencies.

sudo apt update
sudo apt upgrade
sudo apt install runit build-essential git zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake nodejs graphviz

Besides Ruby, Huginn also needs Bundler, Rake, and Foreman. On Ubuntu 16.04, the following command installs these tools:

sudo apt install ruby2.3 ruby-bundler ruby-foreman ruby2.3-dev rake

Debian 8 and Ubuntu 14.04 only offer Ruby 2.1 and 2.0 in their repositories, so you need to build Ruby and the required helpers from the source code (Listing 2); however, this step also places the responsibility for updating Ruby on your shoulders. The commands in Listing 2 create a temporary directory, fetch the source code for Ruby 2.3.4, compile the code, and install the Ruby environment. The final command installs Rake, Bundler, and Foreman.

mkdir /tmp/ruby && cd /tmp/ruby
curl -L --progress http://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.4.tar.bz2 | tar xj
cd ruby-2.3.4
./configure --disable-install-rdoc
make -j`nproc`
sudo make install
sudo gem install rake bundler foreman --no-ri --no-rdoc

Huginn does not support version management such as RVM, rbenv, or chruby. If you do use a version manager, you can expect hard-to-identify problems.

Huginn comprises the web interface and a matching background service that retrieves the information. For security reasons, you can use the command:

sudo adduser --disabled-login --gecos 'Huginn' huginn

to create an account for the huginn user.

Storage Space

The next missing ingredient is the database. The following command installs MySQL on your computer:

sudo apt install -y mysql-server mysql-client libmysqlclient-dev

MySQL has an all-powerful root database user who can tweak the complete database. During the installation of MySQL, you will want to devise the best possible password for this user and type it twice in succession. Once the MySQL packages have been imported, run the command:

sudo mysql_secure_installation

After a few prompts, the script that is launched saves the database. After entering the password for the root MySQL user, proceed to say "no" to the first prompt (about changing the password) by pressing n . However, confirm all other prompts by pressing y . When the script is done, you need to grant Huginn access to MySQL in the MySQL command-line client, which you launch with the command:

mysql -u root -p

To log in, again use the password for the root MySQL user, and at the prompt, enter the commands from Listing 3, which create a user account for Huginn in MySQL; select InnoDB as the storage engine; and allow Huginn to access a database named huginn_production. Replace the <secret> in the first command with a password that is as difficult to guess as possible; then, check whether MySQL has created the new user account correctly by entering:

sudo -u huginn -H mysql -u huginn -p -D huginn_production

CREATE USER 'huginn'@'localhost' IDENTIFIED BY '<secret>';
SET default_storage_engine=INNODB;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `huginn_production`.* TO 'huginn'@'localhost';
exit

When you type the password for the huginn user (<secret> in this case), you should see an ERROR: 1049 Unknown database error. Otherwise, something has gone wrong creating the account.