As a part of my integration tasks, I had the following sequence of systems to deal with:

Sequence of systems handling login

This is the chain of applications that handle a single-signin for a user. At one end we have the Member Management system, written in ASP Classic, that acts as the main authentication system. For various reasons we had to use that system as the main login system.

Once a user logs in, a HTTP request is sent to the Xaraya CMS, along with user details, and a custom CMS module handles the creation of a new user, or the update of an existing user, and logs that user in.

Further down the line is a Vanilla forum application. When the user logs in or out of Xaraya, an event trigger will then send user details to Vanilla (where the user is created or updated) and then logged in.

Logging out works in just the same way, except a different message is passed down the chain.

The result of a user logging in at one end, is that they will automatically be logged into all the applications in the chain. Given a choice, I would have used something else that all the applications authenticate against, but we were tied to various technical limitations.

One specific action that needed to be carried out, was the passing of cookies down the chain, from one application to another. Each application needed to accept cookies from the browser (where available) and also needed to pass new or updated cookies back to the browser. The browser only interacted with the Member Management System, and so the cookies needed to be passed directly between the applications.

To complicate things more, we have a mix of ASP and PHP, and two separate servers. The servers shared a single domain though, so we were easily able to share the cookies across the servers.

One particular step is the passing of cookies across the ASP system. If we ignore the cookies consumed and generated directly by the MMS, then we have cookies that pass through in both directions.

To handle the flow from left-to-right, the following code is used. First initialise an object to send the HTTP message to the CMS (the message to say “log me in”). We are using SOAP to communicate with the CMS.

  ' XMLHTTP object used to send the outgoing message (to the CMS)
  set xmlhttp = server.CreateObject("MSXML2.ServerXMLHTTP")

  ' Set up the destination and headers; open the connection
  xmlhttp.open "POST", "http://example.com/soap/login", false

Now look through all the cookies that have been sent to *this* page, and pass forward any that are of interest further down the line.

  ' Work through a list of cookies that need to be passed in.
  ' Other cookies (e.g. for forum sessions)
  ' may also be available here and may be useful in logging in and out,
  ' helping to retain sessions across logins.
  ' Loop through all cookies supplied by the browser.
  cookieCount = Request.Cookies.Count
  If cookieCount > 0 Then
    For kk = 1 To Request.Cookies.Count
      cookieName = Request.Cookies.Key(kk)
      cookieValue = Request.Cookies.Item(kk)
      If instr("XARAYASID|lussumocookieone|lussumocookietwo|LussumoUserID|vanillasession",
          cookieName) then
        ' If the cookie is in the list to pass on, then do so.
        xmlhttp.setRequestHeader "Cookie", cookieName & "=" & cookieValue
      End If
    Next
  End If

Finally, after creating the SOAP payload, send the request to log in.

  xmlhttp.setRequestHeader "MessageType", "CALL"
  xmlhttp.setRequestHeader "Content-Type", "text/xml"

  ' Send the message and wait for the response
  xmlhttp.send(SOAPpayloadBody)

  ' wait for response
  ' xmlhttp.waitForResponse(3)
  While xmlhttp.readyState <> 4
    xmlhttp.waitForResponse 200
  Wend

That gets the local browser cookies for this domain, passed on to the CMS. Note we only pass on those cookies that will be of interest, i.e. Xaraya and Vanilla Forum cookies in this case.

Now we need to handle the cookies coming back from he right, as we need to pass them back to the browser. After waiting for the response, it is handled as an XML SOAP message.

  ' Anything other than 200 means error.
  If err.number = 0 And xmlhttp.Status = 200 Then
    ' Collect the data returned.
    Set xmldom = xmlhttp.responseXML
    ' Process the returned data...

Now we are able to look through the cookies that have been returned to us.

    '  We now want to sift through any cookies sent back and pass them on to the
    ' current browser. This means extracting the cookies from the response to the
    ' Xaraya login request, and inserting those same cookies into the current page.

    ' Get the response headers.
    ' This is where we would find the returned cookies.
    strHeaders = xmlhttp.getAllResponseHeaders()
    ' Extract the cookies from the headers
    hArr = split(strHeaders, vbCrLf)
    for kk = 1 to ubound(hArr)
      if instr(hArr(kk), ":") > 0 then
        if left(hArr(kk), instr(hArr(kk), ":")-1) = "Set-Cookie" then
          ' Get the cookie name and value (we will include the path and domain
          ' as part of the 'value').
          ' The name is between the first ':' and the first '='. The value is
          ' everything after the first '='.
          cookieName = trim(mid(hArr(kk), instr(hArr(kk), ":")+1, instr(hArr(kk), "=")-instr(hArr(kk), ":")-1))
          cookieValue = trim(mid(hArr(kk), instr(hArr(kk), "=")+1))

          '  Include the list of cookies that you want to be passed on here.
          if instr("lussumocookieone|lussumocookietwo|LussumoUserID|vanillasession", cookieName) then
            ' Send the cookie on to the current browser (unchanged, i.e. same path and domain)
            Response.AddHeader "Set-Cookie", cookieName & "=" & cookieValue
          end if
        end if
      end if
    next

That’s it. All cookies in that list will be returned to the browser, so if any have been created in the process of logging into the CMS or the Vanilla forums, then they will be passed back and upon visiting those applications the user will find they are logged in.

In a similar way, when the user logs off, sessions are cleared (or flags are set in the sessions to indicate the user has logged off) and some of the cookies are cleared by the user of dates in the past.

I have omitted some of the code for clarity, specifically the declarations of the variables, but the guts of the process is here, and I hope it will prove useful to others. If you need any of this explained or expaned, please ask in the comments section.

Follow me on Twitter

This search has gone on for years. I am looking for a newsletter system that is easy to install (to most likely LAMP-based), allows sign-up to multiple newsletters, and is reliable and flexible.

The most important thing is that once configired, I need to be able to hand over the keys to a client so that they can manage it from that point on. That is the hardest part, because no matter what I try, there are always highly technical aspects to sending out a newsletter that really stump the non-technical clients.

This is what I have come up with (and tried) so far:

PHPlist (Tincan)

PHPlist Logo

This has some lovely features, but some pretty dire ‘gotchas’ that can trap an unwary user. For a start the admin screens are as ugly as hell, and the terminology can make it hard to fathom out what each option does (for example, to create a draft newsletter, you need to use the ‘send a message’ option). It is just not intuitive.

Problems for end users involve its uncanny ability to drop stylesheets when editing templates. Another is signing lists of imported users up to text-only e-mails even when ‘HTML’ is selected. Yes these can be worked around, but only if you know the problems exist.

On the plus side, the e-mail sending system is brilliant. You queue messages to send, and leave it to do its job. If it fails (and anything that involves a browser window staying open will fail) then you just start it again and it carries on from where it left off. It keeps track of who a newsletter has been sent to, so you can requeue a newsletter every day until the next newsletter is ready, and it will send just to new subscribers.

The ability to pick of bounces from an IMAP or POP3 mail box is also great. Each bounce will increment a counter, and subscribers can be disabled when a preset number of bounces is reached.

This system also supports tracking, counting each individual e-mail as it is opened, using an image embedded in HTML e-mails.

Another downside is the lack of APIs, making integration difficult. Signing users up to a list automatically, while they are signing up to a CMS, is impossible without some major hacking.

The problems with PHPlist stem from it being around a long time. It has grown and grown over the years, but is creaking at the seams. Everything it does is great, but the way it does it is not so hot. It is time, IMO, to throw away the old code and start again, with all the same features, an established framework (e.g. Zend or CakePHP) and it will be a killer application.

I must also add that I believe PHPlist got the data schema right from the start. Most other newsletter systems I have looked at have gone for a much more simplistic approach, that makes it much harder for them to advance their products beyond very simple functionality.

With all that it is, I do use it, and find it great at what it does. What I can’t do, howver, is roll it out to clients, because it results in nothing but hassle from the non-technical users who just can’t get to grips with its mix of high-level features, and low-level ‘black art’ knowledge that is needed.

poMMo

poMMo logo

This one has to get the award for ‘hardest to find mailing list manager’. It was formally the bMail project and is hosted on SourceForge. Trying to find it when you can’t quite remember its ‘web 2.0′ name is quite an effort, but the result is well worth it.

Checking the project’s subversion repository, it does not seem there has been any activity on this project since August 2008, which is not very encouraging.

First of all, the admin screens are lovely. They are smooth, AJAX/jQuery-based, clear and well designed. The whole thing feels smooth – it works with you.

The first thing to note is that it does not support multiple mailing lists. In fact, it does not support any ‘mailing lists’ at all. However, what it does provide is a means to mail out to groups of users depending on attributes of those users.

For example, you could provide a series of checkboxes for a subscriber to select what subjects they are interested in. When sending out a newsletter, the author would send it to all subscribers who have expressed an interest in the subject of that newsletter. Now, that does sound rather like a multiple newslist system, but the subtle difference is that there is no ‘newslist’ object in the database – anything that looks like a newslist subscription is specified by the administrator, and not enforced as any kind of fundamental part of the system’s structure.

I think that approach works well. The system is simpler, and the flexibility is increased. There are a few downsides though.

The first downside is that without a newslist for a user to subscribe to, there is no place in which you can find out exactly when a user subscribed to that newslist. That means sending out additional copies of a newsletter to late subscribers simply cannot be done. Remember PHPlist knows who subscribed to what and when, and so is able to send out ‘catchup’ newsletters so a subscriber will always get the latest newsletter very soon after subscribing.

The other downside is in organisation. Without being able to group newsletters into newslists, it is much harder to provide archives organised by subject. On the other hand, the archives are there, with very easy access to the newsletters from a web-based front end.

The e-mail sending backend is both genius and a little frightening for a control-freak such as myself. It also misses a few tricks, I think.

In order to send e-mails, the system first creates a list of who to send to (in a text file, I believe). It then spawns a process through the HTTP protocal to send as many e-mails as it can in the PHP timeout period (you would set this to as long as possible). Before that script times out, it spawns another process to carry on where it left off, and so on until all the e-mails have been sent, whereupon a lock file created right at the start is released.

This is genious in that the developers have worked out a way to run a background script that can keep going even when the browser window is closed. It is frightening for much the same reason. I just don’t feel comfortable letting something like that loose on my server. It could be running all night in some kind of endless loop, and I would never know until I get irate e-mails from subscribers saying that I have filled up their inboxes.

Another minor flaw in the e-mail sending is that no record is kept of who each newsletter has been sent to. Doing that would make it very easy to sort out who has received a copy and who needs a catch-up copy sent. Following from that, without records of e-mails sent, there is no place to hang any flags to say whether that e-mail has been opened and read, so e-mail tracking is out. Clients need to know these things: how well read are those newsletters? Everyone has a master up the chain to report to, and someone up that line, often holding the purse-strings, likes simple measures of performance.

My ideal newsletter system would contain the database and functional features of PHPlist, with the administration front-end of poMMo.

Other Systems

I’ll add a few more when I get the time. I’m mainly looking at Open Source newsletter systems, aimed at sending to self-subscribed users (i.e. not bulk spam systems). Integration with an existing site or CMS is high on the priority list, along with ease of use for non-technical users. Other systems we will be evaluating are:

• ListMessenger – Free light version, but dirt cheap Pro version with all the features you would want. This one seems to include captchas for registration.
• Dada Mail – Old, well known, Perl-based, and it looks like a pain to install.
• Sympa – Again, Perl. This one seems to have been designed by engineers. That is to say the system looks very robust and complete, but there is little gloss to the system. Like Data Mail, you need root access to install it, so you have to be careful about dependances with other modules on your server.

Follow me on Twitter