Monday, 22 November 2010

EEE: erlang inets http authentication with mnesia

Again, I am using R13B04 (erts-5.7.5) on Fedora 13.

Unfortunately erlang's inets http server only performs Basic authentication.  So I would only authenticate over https, like I've shown here.  Setting up erlang inets for SSL can be tricky depending on what you want to do.  One important gotcha is detailed in this post.

Create Tables

First, your application must create some mnesia tables for authentication. For this you need the record definitions.  The documentation for mod_auth says to add the following line to your module.

-include("mod_auth.hrl").

erlc failed to find the file, so I found it in "/usr/lib64/erlang/lib/inets-5.3/src/" and copied it to my project manually.

Creating the tables is no different to the documentation

mnesia:create_schema([node()]),
    mnesia:start(),
    mnesia:create_table(httpd_user,
                        [{type, bag},
                         {disc_copies, [node()]},
                         {attributes, record_info(fields, 
                                                  httpd_user)}]),
    mnesia:create_table(httpd_group,
                        [{type, bag},
                         {disc_copies, [node()]},          
                         {attributes, record_info(fields, 
                                                  httpd_group)}]),
    mnesia:wait_for_tables([httpd_user, httpd_group], 60000).

The documentation says this is a naive implementation, a trick I have used is to catch the output of mnesia:create_schema.  Ok means that the schema didn't exist before and the tables have to be created.  If there is a better way, I'd like to hear it.

Add user and group

Next, adding a user and group to the database, I use the mod_auth functions

true = mod_auth:add_user("superuser", "password", "Super User", 443, "/test"),
true = mod_auth:add_group_member("users", "superuser", 443, "/test").

Configuring for Authentication

Below is the next version of my "443.conf" file which includes the necessary configuration for authentication.

[
 {modules, [
  mod_alias, 
  mod_auth, 
  mod_esi, 
  mod_actions, 
  mod_cgi, 
  mod_dir, 
  mod_get, 
  mod_head, 
  mod_log, 
  mod_disk_log
 ]},
 {port,443},
 {server_name,"localhost.localdomain"},
 {server_root,"log"},
 {document_root,"secure"},
 {erl_script_alias, {"/test", [test]}},
 {directory, 
  {"/test", [
   {auth_name, "Data Server"}, 
   {allow_from, all}, 
   {auth_type, mnesia},
   {require_group, ["users"]}
  ]}
 },
 {socket_type, ssl},
 {ssl_certificate_file, "localhost.pem"},
 {error_log, "error.log"},
 {security_log, "security.log"},
 {transfer_log, "transfer.log"},
 {mime_types,[
  {"html","text/html"},
  {"css","text/css"},
  {"js","application/x-javascript"}
 ]}
]. 

To access my test esi module, I now need a to use a password.

EEE: Binding erlang ssl to ports < 1024

After a bit of hair pulling, I finally worked out how to get erlang's inets to bind to port 443 and service https requests. Again, I am using R13B04 (erts-5.7.5) on Fedora 13.

I found this page on erlanganswers.com which said to use setcap on Linux > 2.6.24. Unfortunately this didn't work. After clearing the hair off my keyboard, Running "netstat -ptl" while temporarily serving https on port 8080 showed me that beam was not doing the listening, but ssl_esock.

setcap "cap_net_bind_service=+ep" /usr/lib64/erlang/lib/ssl-3.10.8/priv/bin/ssl_esock

No need to setcap any other file except this one.

Below is a "443.conf" file that configures inets httpd to serve https only. Look at my previous post on starting inets http server for the commands to start it.

[
 {modules, [
  mod_alias, 
  mod_auth, 
  mod_esi, 
  mod_actions, 
  mod_cgi, 
  mod_dir, 
  mod_get, 
  mod_head, 
  mod_log, 
  mod_disk_log
 ]},
 {port,443},
 {server_name,"localhost.localdomain"},
 {server_root,"log"},
 {document_root,"secure"},
 {erl_script_alias, {"/test", [test]}},
 {socket_type, ssl},
 {ssl_certificate_file, "localhost.pem"},
 {error_log, "error.log"},
 {security_log, "security.log"},
 {transfer_log, "transfer.log"},
 {mime_types,[
  {"html","text/html"},
  {"css","text/css"},
  {"js","application/x-javascript"}
 ]}
].

EEE: Configuring Erlang inets

Erlang's built in web server (part of inets) has a number of quirks, first one I came across is listing the server modules in a specific order in the configuration file.  I am using R13B04 erts-5.7.5 on Fedora 13 for this post.

Much of this came from this stackoverflow question.   I've almost copied it completely except for a few details below.

Here is a complete configuration file example. (8080.conf)

[
 {modules, [
  mod_alias, 
  mod_auth, 
  mod_esi, 
  mod_actions, 
  mod_cgi, 
  mod_dir, 
  mod_get, 
  mod_head, 
  mod_log, 
  mod_disk_log
 ]},
 {port,8080},
 {server_name,"localhost.localdomain"},
 {server_root,"log"},
 {document_root,"www"},
 {erl_script_alias, {"/test", [test]}},
 {error_log, "error.log"},
 {security_log, "security.log"},
 {transfer_log, "transfer.log"},
 {mime_types,[
  {"html","text/html"},
  {"css","text/css"},
  {"js","application/x-javascript"}
 ]}
].

Here is how to start inets using the configuration file.

 inets:start(),
 inets:start(httpd, [{proplist_file, "8080.conf"}]). 

Make sure that both "www" and "log" directories are there otherwise it won't work either.  If you put a test file in the "www" directory, opening http://localhost:8080/test.html will download it.

A few things didn't work, so I wouldn't bother trying this
  1.  Trying to add extra configuration items before or after proplist_file when calling inets:start.  I tried to specify the port dynamically in this fashion, but it didn't work. Looking at the source might help me work out why.
  2. Switching port 8080 to a port < 1024.  Linux doesn't let you do this without some extra trickery.  I'm investigating how to get around this.

Monday, 1 November 2010

Instant SCADA

SCADA product manufacturers such as Allen-Bradley, Siemens, Honeywell etc. each try to distinguish themselves in the market place with their own proprietry protocols and extensions to industry standards. The benefits of connecting disparate products together have given them incentive to join together and standardise data collection and HMI protocols. These standards were based on the middleware systems available at the time. In my opinion, the increasingly popluar Extensible Messaging and Presence Protocol (XMPP) should be the basis fo the next dominant internet middleware and hence the next technology for connecting disparate SCADA systems. This post discusses some history of middleware in SCADA and how I came to believe that XMPP should be next.

Microsoft OLE -> Microsoft DCOM -> OPC

The OPC foundation is a collaboration of industry partners and is the steward of the Object Linking and Embedding (OLE) for Process Control (OPC). The purpose of the original OPC spec was to provide a standard method of connecting SCADA equipment to personal computers running the Windows operating system. OLE and COM were the Windows way of making re-usable, object oriented components that could be joined together to make applications. These components (a COM object) often had visual representations and could be programmatically glued together to make more complex applicaitons. DCOM or Distributed COM is the way of bundling up the state of the COM object and transferring it across a network. In OPC, the state of a process component is bundled into DCOM representation and the COM object is transmitted to a Windows machine or SCADA equipment. OPC leveraged off a popular technology at the time of it's inception.

HTTP -> SOAP -> OPC Unified Architecture

Next is OPC Unified Architecture (OPC UA). In people's day to day lives, the importance of the PC for storing and processing data is now surpassed by the internet. The OPC Unified Architecture (OPC UA) brings process automation into the internet age by being built upon standard internet technologies. The internet is not a homogenous environment like Windows so a prioprietary microsoft only technology is not adequate. OPC UA has thrown out COM an DCOM and replaced it with Simple Object Access Protocol (SOAP). SOAP is the Microsoft sponsored, open successor to DCOM. It is based on the Extensible Markup Language (XML) which is not easy to implement on embedded systems, so OPC UA also has a parallel binary protocol.

Human eyeballs -> Middleware -> SCADA

One obvious trend is that the previous incarnations of OPC were based on Microsoft sponsored technologies, but the Internet has levelled the playing field and Microsoft tech is no longer a pre-requisite for its adoption. The trend that this taxonomy proves is technology originally designed for data dissemination to humans has become the basis for system to system middleware. SCADA is an application of middleware and the trends in middleware become the trends in SCADA.

The current trend in middleware is messaging. The reason why there is a shift from REST protocols, like SOAP, to message based ones is the peer to peer architecture. Peer to peer architectures decrease latency and messaging overhead when compared to client server based architectures. Decreasing latency makes processing faster, important for fast paced processes such as stock trading. Decreasing overhead is important for power consumption.

XMPP was previously known as the Jabber protocol and designed to be an alternative to proprietary systems such as ICQ, MSN and Yahoo. It is an open IETF standard protocol for instant messaging. Facebook chat,  Google talk and Google Wave are all based on it.  Why I think XMPP will become the next dominant middleware instead of other messaging protocols such as AMQP or ZeroMQ is the same reason why SOAP and not CORBA, JMI or RMI was chosen for OPC UA. Education and resources. Developer education in writing dynamic web applications was easily transferable to SOAP and infrastructure that serves web pages also serve SOAP remote procedure calls. For XMPP,  chatting teaches you the concepts of messaging and XMPP software is proliferating and maturing as more and more people instant message rather than email.

Monday, 3 December 2007

Use git-svn Eric/Erik!

Geeky types love to debate. And source control is one of those topics geeks debate about non stop. I am no different. At every new job I've had, I've always battled with crap source control tools (or with none at all) and argued with managers to at least start some change for the better.

I came across this: Dear Eric... you should use git-svn

I wish Erik (with a k), the head maintainer of a project I often work with would adopt git.

Friday, 8 June 2007

GPLv3, DRM, and Tivoisation simplified

DRM is about restricting access to digital information. It's like building a digital fence around your movie, song or program. Once that fence is built, what's inside can be protected against modification, kept secret and/or admission can be charged.

The makers of the Tivo product have used DRM to erect fences around their products and the GPLv2 software that runs on them. Many authors of GPLv2 software are upset at this, because they see it as restricting access to property they donated for public use. But alas, there is nothing they can do. When GPLv2, was written, fences like DRM weren't invented and this possibility was not forseen. GPLv3 is required to fix this loophole. It explicitly states that, if you are not the copyright holder of a piece of software, you are not permitted to erect a DRM fence around the software without supplying the means to access it (the key to the gate).

What has this got to do with my music and high definition DVDs? Not much really. Publishers own the content and they can distribute it any way they want. If you don't like the DRM restrictions then there are other entertainment options out there. You should probably try them.

Monday, 4 June 2007

Fedora 7 Review

After looking at Ubuntu 7.04 (Fiesty Fawn) and deciding to replace it with Fedora 7 as soon as it come out, I am happy to report that I've kept my word. Here are my first impressions.

The good:
  • 1 CD sized download with all the good stuff.
  • Same Gnome goodness I've come to expect from all distributions.
  • RAID and LVM support in the installer.
  • My "fake RAID" is detected by the installer!
  • rpm.livna.org is ready with audio and video codecs
The not so good:
  • Not sure I like the new desktop theme (Hot air balloons?)
  • My PPPoE internet connection won't automatically start. (I'll have to find out why)
Verdict:
A couple of hiccups, but not a completely bad experience. The switch to a single repository, as far as users are concerned is seamless. The creation of a installable Live CD I'm sure was because of "market pressure", but this option is very welcome. I've researched a little bit about what needs to be done to make my own Live CD from the fedora repository and at the moment it seems like it will be a lot of fiddling with rpm and configuration files. Building a Live CD with my photo as a the Desktop banner for my mum is not a click and go process.

Tuesday, 8 May 2007

Ubuntu 7.04 (Feisty Fawn) Desktop edition

The first GNU/Linux distro I installed was Slackware of 3.5" floppy disks. I was awestruck by that first GNU/Linux expereience, especially when it told me that my old Pentium 60 Mhz had the "widely publicised" fdiv bug. I'm not really a fanboy type of guy and since then, I've chopped and changed distributions a little. I've used Redhat, Mandrake, SUSE, Knoppix, Fedora Core (3, 4, 5 ,6) Debian and even Gentoo. Debian and Gentoo honestly don't do anything for me as I've done those fiddly things a million times already. Just not interested in doing them again. I want something that just works. But my kind of "just work" is different to Ubunto's.

What is good about Ubuntu.
  • I didn't spend hours downloading a DVD iso or 5 CD iso files
  • I can play with it before installing it
  • I can boot the install CD and use it as a rescue tool WHEN I screw things up
  • I got used to sudo, and kind of like it now.
  • It booted fast, and it's pretty.
What is bad.
  • No LVM or RAID support in the installer
  • Had to search for pppoeconf to get connected to the Internet
  • Network configuration GUI has no pppoe support
Verdict:

My end goal has never been a Linux desktop at home. I want a cool high performance workstation that I develop on and surf the net . At times I felt like giving up on Ubuntu and just install FC6 because when I wanted to something a little uncommon, like software RAID or PPPoE internet connection, it wasn't simple or straightforward. While you can play with Ubuntu before installing, it's not knoppix.

Ubuntu seems to be for the "for dummies" crowd. For me, it doesn't cover enough of the functionality that I have come to expect from a good distro. Fedora 7 will have a similar installation model to Ubuntu, sudo is already available on Fedora Core and the livna rpm repository has non-free video/sound codecs that I could use if I wanted too. All the things I liked about Ubuntu will be in Fedora 7.

When Fedora 7 is released it is likely that Ubuntu will be wiped from my computer and the CD I burned will be thrown in the bin.

Sunday, 22 April 2007

A desktop computer RAID

Haha, cheezy title.

Don't get it? My desktop computer did not get subpoenaed by the Record Industry Association of America. I am referring to redundant array of inexpensive drives (RAID). Traditionally used only on mission critical and high performance servers, RAIDs provide data protection through redundancy and higher performance through parallelism. But improved data security and higher performance are key sales points for desktop computers so now the technology is increasingly available on PC's. Dell has provided RAID options on some of their desktop PCs for close to 3 years now.

A modern desktop PC usually has RAID capabilities in its SATA chip set. When the computer boots, a BIOS tool is used to set up a RAID configuration. This is not a "real" hardware RAID but software RAID implemented in the BIOS, also known as a "fake RAID". Windows 2000/XP use BIOS routines to access data on the RAID until you install the "Application Acceleration" drivers provided by Intel or whoever the motherboard vendor is. It will then use those software routines to access the RAID devices.

Linux has had software RAIDs for a while but their software RAID system is incompatible with proprietary, BIOS based "fake RAIDs". While Linux "fake RAID" drivers exist and are improving with time, in my opinion, a Linux software RAID is a better choice if Linux is your primary OS. Linux software RAIDs are completely open, will work on any "normal" hard-drive, will co-exist with other operating systems. Tools for provisioning, backup and recovery are all in the Linux environment and they are all well tested. The only disadvantage is the incompatibility with "fake RAIDs". Hard drives cannot be "normal" and a member of a "fake RAID".

The Linux Software RAID Howto is a good resource to begin with if you want a deeper understanding of RAID on Linux.

Friday, 30 March 2007

Gtk+ vs Qt

If you are developing a GUI application for a Unix OS, selecting a toolkit is a trickier decision than you might think. The two main desktop environment contenders these days are KDE and Gnome. Both are nice clean interfaces whose latest versions are well equipped to handle any UI problem that you will throw at them. Underlying each of these are windowing toolkits used to present the respective user interface elements. Gtk+ for Gnome and Qt for KDE. Which desktop environment do I target and which toolkit do I use?

Qt (on which the KDE desktop environment is built on) is owned by Trolltech. Trolltech maintains strict control over the Qt framework making it a Trolltech product. You become a Trolltech customer if you use Qt. Qt and it's derivatives are commercial products which Trolltech releases under a GPL only license. This means if your product uses Qt it must be GPL or (or an FSF approved license) or you must pay Trolltech licensing fees.

Gtk+ (on which GNOME is built upon), is a community project with many commercial and volunteer members. It's being developed under the GNU umbrella and is LGPL licensed. This means you don't have to pay licensing fees to use it commercially and you are free to open or close source your project at your discretion. If you find yourself making additions to the Gtk+ framework, you can submit them to the community for review and they may be accepted.

If your project is purely commercial then both are acceptable choices. Trolltech will happily help you if you pay a licensing fee, and a Gtk+ community member may help you if you pay them for their time. If your project is purely open source than both are also acceptable. The problem comes if your project could be comercial or it could be open source or it could be something in between then Gtk+ becomes the only choice.

Tuesday, 30 January 2007

Fonts!

I need free asian fonts for a project I'm working on. In my travels I've come across a few cool things.
  1. Mojibake. Now you know what to call it when you surf to a foreign language web page and your computer screen shows gobledeygook.
  2. Wikipedia help on getting east asian fonts to work.
  3. The Unicode font guide. Discussions on what font for which language.
  4. Freedesktop.org's CJK font page

Tuesday, 9 January 2007

GPL for libraries

Step 1. Create a code library and license under the terms of the GPL.
Step 2. Encourage adoption among students and other open source products, but keep ownership of copyright. (have all contributors assign copyright to you instead of the GNU foundation or keep it themselves)
Step 3. Secretive use of your code requires commercial license = Profit.

Examples: Trolltech Qt, MySQL, BerkelyDB(now owned by Oracle).

RMS, software's Confucius.

In 1991 Richard Stallman (or RMS) created version 2 of the GNU General Public License . It is my personal belief that by writing this 18KB of American English he has done more for software development than anybody who has written 18MB of code. He is to software developers as Confucius is to Chinese society. I'm looking forward to the final version of GPLv3 and working on or with a project that uses it.