This site makes very heavy use of JavascriptIf you don't have Javascript turned on, it is not going to work very nicely at all.

Multi-column layout filling all available space

I've just had my life revolutionised by a deeper understanding of a particular part of the CSS spec. It looks like magic but it isn't.

Consider a two-column page layout, where the left column needs to be variable width (up to a maximum), wrapping its content, and the right column needs to fill the rest of the horizontal space.

<div class='left'>stuff</div>

<div class='right'>right stuff</div>

</div>

If you float the elements (one or both), you get both blocks being the minimum size they can be. Same if you inline-block.

It turns out that what one needs to do is have the .right div start a new Block Formatting Context (BFC).

A BFC will respect the edge of floats and will have its children block elements fill its full width.

A BFC will also adjust its height according to its children, even if its children "float" - so no more messing around with "clear".

My two elements would normally be in the same BFC, and so if I 'float: left' the .left element, the .right element will need to be given a width or it will take 100% of the document width and will "drop down" below .left.

By putting .right in its own BFC, it will naturally take up 100% of the remaining space next to the float:left .left element.

How can you activate this, you may ask? Simply put "overflow: hidden" on .right :

.left {

max-width: 33%;

float: left;

}

.right {

overflow: hidden;

}

ALL credit for this goes to this stack overflow question/answer:

http://stackoverflow.com/questions/1260122/expand-div-to-take-remaining-width

And to this article:

http://colinaarts.com/articles/the-magic-of-overflow-hidden/

And, of course, to the CSS spec:

http://www.w3.org/TR/CSS21/visuren.html

...Click for More

Article

Block Formatting Context

CSS

Fill available space

Float left

full width right column

Multi-column layout

Programming

File scrambling for stronger encryption

I've been thinking a lot about encryption recently and what makes it easy to crack (or not).

One thing which can make it easy to crack is knowing part of the "plaintext" - the thing that was being encrypted. If one knows a bit of the plaintext and one has the encrypted text, it may be possible to work out how to get from "plain" to "encrypted" - i.e., work out the key. And once you have the key, you can decrypt the rest of the message.

Mostly, this isn't that easy, as a modern cipher will operate on blocks of plaintext in such as way that you can't point at a single byte and say "this has been changed to this other byte, therefore the key must be XXX". But any bits you know of the plaintext might give you clues, reducing the number of possible keys from billions to thousands, thus helping a "brute force attack" immensely.

So if there are artefacts in the plaintext, it can potentially be broken more easily. Why does this even matter? Because data transmitted by computer has all kinds of artefacts in it.

Artefacts: The enemy of encryption

When you visit Google in your web browser, your computer makes a request and Google responds. As the request is transmitted from your computer, through the myriad of devices on the internet to eventually reach a Google server, it has various things added to it to tell Google's servers many things that it may find useful when responding: Your browser type, if your browser supports encryption, the kind of character set to use, and the very first few bytes of the request (and response) are (almost) always "HTTP/1.1".

So all of this data in the request is predictable, if not completely static. When Google replies, the HTML it uses is also very static. If it is a public website to which the attacker has access, most of the page that they see will be identical to what you see. It might only be a few of the numbers on the page that are actually different between their visit to the site and yours. As much as 90% of the plaintext being encrypted and sent to you is therefore known to the attacker. That's a whole lot of artefacts.

Finding artefacts is the basis of the W.E.P. (Wireless Encryption Protocol) attack: It was found that Wireless Access Point that encrypted with WEP always responded with a similar message when a particular kind of duff request was presented to it. A carefully written "attack program" would fire millions of messages at an access point in a minute or so and analyse the results. Analysing these results and knowing what the plaintext was, the "attack program" was able to very quickly present you with the WEP key. Hey presto, broken wireless.

This is a greatly simplified explanation of the WEP attack, which is much better described here: http://en.wikipedia.org/wiki/Wired_Equivalent_Privacy

Digital TV Pay-per-view is encrypted. The Wikipedia article makes some interesting points on how to brute-force attack the key, and also how to use "rainbow tables" to speed up the brute force:

http://en.wikipedia.org/wiki/Common_Scrambling_Algorithm

So, in the case of an HTML document, if we assume that most of the plaintext is known, how can we protect the unknown, sensitive bits - and our encryption key?

This is where some simple file scrambling can come in to its own: An attacker may know a lot of your insensitive plaintext and will have captured the encrypted text, but if they don't know where in the file or network packet each is, this knowledge can no longer help them recover the key or the sensitive parts of the data.

Scrambling the data to remove artefacts

What do we mean scramble?

Very simply, to randomly move characters around so that a predictably laid out file with predictable contents appears to be a random stream of bytes.

Let's roughly scramble some short plaintext:

EQUATOR

OEAUTRQ

So this hasn't been encrypted, it's just been randomised. We need to be able to un-randomise it to get back to our plaintext, so we need to know how the letters have moved. We'll put some numbers on the front to specify where each letter came from and is now.

There are 7 letters, so we'll need 7 numbers on the front. The position of each number refers to the position of the letter we moved. So the first number refers to the first letter, the second to the second, and so on. The actual number itself refers to where that letter has moved to. So if the first number is 3, it means the first letter is now in position 3.

Let's see how that looks with the above:

2743516OEAUTRQ

So we have some random numbers which can't be known sandwiched on to the front of some plaintext where one or more of the letters have been previously guessed.

But now our attacker doesn't know where those guessed letters are. Let's encrypt. Just for demonstration purposes, we'll use a Caesor Cipher. Here's the translation table:

ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890

90ABCDEFGHIJKLMNOPQRSTUVWXYZ12345678

So our new ciphertext, based on our scrambled plaintext, is:

49657382GCWVTO

Let's assume the attacker somehow knows the first and last letters of the plaintext are "E" and "R", respectively.

Trying to use this knowledge, we fit the plaintext E to the ciphertext G and the R to the O. This would give us a key of 2 in the first case and -3 in the second, which can't be right! So the attacker is denied using knowledge of the plaintext to find the key.

Earlier, we talked of HTML. If an HTML document had been scambled and then encrypted, it would not be possible to say that the first 4 letters of the document before encryption was HTTP. They may know up to 90% of the plaintext, but from their point of view what was actually given to the encryption algorithm is effectively gibberish.

The artefacts are gone.

A potential file format for artefact-free scrambling

How would we scramble a file real world without leaving any artefacts for an attacker to use? We surely can't go adding a byte for each byte in the file! It would be huge, and would need compressing, which will leave artefacts in the file which can be used to recover the key - the same kind of artefacts we are trying to be rid of by scrambling.

Rather than working on bytes, it should operate on blocks. The blocks should be a random size - the size is picked at scramble time. To pick a rule at random, we'll make the block size 1% of the file size +/- (randomly) up to 2%. This will give us between 33 and 100 blocks. The number of blocks will need to be recorded in our header. That's ok, because it is a random number, unknown to our attacker, so it doesn't create an artefact.

A file is unlikely to be exactly the right number of bytes in length to divide in to an equal number of blocks. So the last bit of the file will be padded with random garbage. Rather than recording the original file size, which may be known to an attacker and thus give us an artefact, the number of bytes that were added for padding is recorded next. This is related to a random number we picked earlier, so is again unknown to our attacker. But it gives us enough information with which to reconstruct the original file.

Then, all the random "this block moved to X" numbers are generated. Each number will be stored in 2 or 4 or 8 bytes, depending on how large the file (or, indeed, network packet) is. The numbers are placed on the front, followed by the blocks in their scrambled order.

So the final scrambled file would look like this:

(Block size)(Padding size)[list of random numbers][blocks in random number order...]

None of the numbers on the front can be predicted and none of the data later on is in the right place, so, if this were then encrypted by any number of (hopefully) secure encryption algorithms an attacker now doesn't know which bits of cipher-text to compare with which bits of plain-text.

Not only is the key infinitely harder to recover now, but brute-forcing is now a more expensive operation: every time a key is tested out of the (say) 50 billion possible keys/passwords, the result has to be tested by unscrambling. This is extra clock cycles and slows the whole thing right down. If testing a single password/key would normally take, say, 10ms, and unscrambling takes, say, 1ms, that is an extra 10% effort. If testing all possible keys takes 10 days, that's an extra day.

One day I'll write a proof-of-concept of this scrambling technique, and maybe even prove that scrambling before encryption is a worthy thing to do. If it is worthy, this simple technique could be transparently built in to many commonly used consumer encryption technologies: SSL, GPG, IPSEC etc could all scramble before encrypting, boosting the average use's protection on the net a million-fold.

Postscript

Here's a good graphical example of how different encryption algorithms work, and how, unless you're careful, the encrypted result can still show major details of what was encrypted: http://pthree.org/2012/02/17/ecb-vs-cbc-encryption/

If the image files had first been scrambled before encryption, both the algorithms demonstrated would result in complete gibberish out.

...Click for More

Article

Brute Force

Encryption

File Scrambling

Programming

PDO numeric values and JSON

Alternative title: Why is PDO converting all my numbers to strings?

When querying a database with PDO in PHP, you will get numeric values returned as strings. This is fine in PHP and you don't need to worry.

If you convert to JSON though and pass to Javascript, you may have issues, as Javascript is a typed language.

Since PHP5.3.3, you can pass JSON_NUMERIC_CHECK to json_encode to have it detect numeric values and cast them as appropriate.

...Click for More

Article

javascript

json

Numeric values

PDO

Programming

UCWords in Javascript

Sometimes, when reimplementing, say, an Access database as a Web-based application, you may get a request such as "My Access app used to be able to auto-capitalise words as I type them!"

Here's how to implement such functionality in Javascript:


(function()
{
    /* Handlers for input.ucwords which auto-capitalise 
     * the field contents.
     *
     * Handles any textinput on screen with class 'ucwords'
     */
    function fn(e, t)
    {
        var el = $(e.target);
        if( !el.is("input.ucwords") ) return;

        // Small timeout so this is multi-use between "blur" and "keyup"
        if( !t ) return setTimeout(function(){fn(e, 1)}, 500);

        var words = el.val().split(/ +/);
        for( var x=0; x<words.length; x++ )
            words[x] = ucword(words[x]);
        el.val(words.join(" "));
    }

    function ucword(word)
    {
        return word.substr(0,1).toUpperCase() + word.substr(1).toLowerCase();
    }

    $(document).focusout(fn);
})()

Enjoy.

...Click for More

Access Conversion

Article

Auto Capitalise words

javascript

Programming

UCWords

SimpleXML with a namespaced SOAP document

Top Tip. If you have an XML document which uses varying XML Namespaces and you want to navigate it using PHP's SimpleXML, you'll need to use 'xpath' to locate your starting node.


<?xml version="1.0"?>
<ns0:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/">
<ns0:Header>
    <AccountInfoHeader xmlns="http://services.hnseu.com">
        <UserName xmlns="">someuser</UserName>
        <Password xmlns="">sompass</Password>
    </AccountInfoHeader>
</ns0:Header>
<ns0:Body>
    <CaseUpdatesRequest xmlns="http://services.hnseu.com">
        <Timestamp xmlns="">2013-07-07T00:00:00</Timestamp>
    </CaseUpdatesRequest>
</ns0:Body>
</ns0:Envelope>

If you feed this in to the SimpleXML or SimpleXMLIterator constructors, you will find you are rooted in the Envelope, but it appears to have no children. You can't navigate to the Body node.

You can feed the "namespace" in to the SimpleXML constructor for the doc, but then you seem to lose the ability to navigate to non-namespaced nodes, and some nodes seem to disappear.

In this example XML, we want to access the Body properties, which are not namespaced.

The trick is to use 'xpath' to get a SimpleXMLIterator rooted at Body. The object returned will be able to see everything underneath in a nice, easy, SimpleXML manner.


$xs = new SimpleXMLIterator($xml);
$bod = $xs->xpath("//ns0:Body");
$upd = $bod[0]->CaseUpdatesRequest;

$bod now represents the Body node, and you can use SimpleXML style addressing to navigate down from this node.

Enjoy.

...Click for More

Article

php

Programming

SimpleXML

SOAP

XPath

Eurohike 210TS Size

Some people on the Interwebs are lovely.

In trying to fix my old but faithful Eurohike tent, I found I have no idea how long the poles are supposed to be. This is a bit of a problem if you have snapped a section and lost part of it. It is even more of a problem when this wasn't even the right pole to start with (It's not the first repair)

Searching around, I could not find any spec that actually tells you how long the poles are. But I did find a chap selling an identical tent on eBay. He very kindly told me what I needed to know:

"Short total length is 1765mm assembled, long total length is 2654mm assembled and compressed as used.Short
 arc has shorter poles than long arc and of course end poles are 
different length to middle section and depends on whether you include 
metal tube. However from total lengths given you can make to suit"

What a legend. Hopefully this will now be useful to someone else too.

Enjoy.

...Click for More

210TS

Article

Eurohike

Pole

Repair

Tent

http://www.youtube.com/watch?v=j23NIWuN5Y4

...Click for More

Episode 1

Funny

Ideas Men

Video

YouTube

http://www.youtube.com/watch?v=VrVYzwXabAM

...Click for More

Ambient

Electronic

Music

New Family

Plaid

Video

YouTube

Stored Procedure PHP integration

I may have mentioned before, I do rather like SQL and stored procedures. They let you work with large datasets, quickly, nice and close to the server, in a naturally secure fashion.

But how can you make them easy and seamless to work with?

Here I will show you a few tricks which will let you call your database API, return results, handle exceptions and even do some logging in the easiest, laziest ways possible.

First off, call-time integration.

Classically, the way you would normally call a Stored Procedure is very similar to executing any SQL statement from PHP:


$sql = "call spSomeProc(?, ?, ?)"
$s = $db->prepare($sql);
$res = $s->execute(array(1, 2, 3));
(if $res is ok, start fetching result sets...)

It's a bit laborious, isn't it.

Let's start with abstracting this bit: Create a class to contain this code, call it "SPWrapper". Make it a Singleton so you can access the same instance from anywhere. This is important, because its Constructor will be capable to connecting to the DB, but we only want a single DB connection.

And now the magic - PHP has what it calls a "magic method" called "__call". Create a public function called __call.

The first argument to __call will be the name of the function that was actually called. Use func_get_args to get all variable arguments. Shift the first item from the array in to $procName.

Now build a "call query" like the above, prepare, execute etc. If $res returns anything but true, find the last error message from the server and throw an Exception with it.

So, now you have API integration and can run any Proc like this:

$sp = SPWrapper::getInstance();
$rSet = $sp->spSomethingDoSomething($a, $b, $c);

Nice.

What about user-defined errors?

Sometimes, you will want to check certain things are correct inside a stored proc and gracefully refuse to cooperate if all is not well. There are limited ways to return errors from MySQL, and fewer ways to trap those errors. So let's stick with what we know and what we can do - returning result sets:

Arrange that when your __call function has received a results set from a proc, it checks its length and columns. If that results set is a single row consisting of an "ErrCode" and "ErrMessage", throw an Exception with that code and message.

This does, however, take a bit of work inside the proc. Not much work, but adhering to this design pattern:

create procedure spSomethingDoSomething(a int, b int, c int)
begin
declare errCode int default null;  declare errMsg varchar(512) default null;
codeblock: 
 begin
  ....
  ....
  if (some condition not met) then
  begin  
    set errCode = 45;  set errMsg = "Some bad error"
    leave codeblock;
  end if;
end;
call spErrHand(errCode, errMsg);
end

So any time you detect an error condition, a simple setting of the code/message and leaving the block will create an exception in PHP which will bubble all the way up through your app until properly handled.

You may be wondering what spErrHand does: It is a convenience function which tests if its arguments are NULL. If not, they are selected out:

select vErrCode as ErrCode, vErrMsg as ErrMsg;

This guarantees consistency and saves much typing.

So now we have the basics of an interface with PHP, the last thing we need is some logging. SQL / Stored Proc programming can be a pain when you can't see what is going on, and a decent logger is your best friend.

Arrange a table to put log messages in. It should be a permanent table, but using the Memory engine for speed. The table itself should have a field for the message itself, an Id for ordering and a field for a "tag".

The "tag" will be a short, random string made up on entry to the proc, or (my preference) made up when PHP starts up. A useful "tag format" I find is AAAAA/BBBBB, where AAAAA is the last 5 chars of the current session token, and BBBBB is 5 random chars. This lets me track a user's session or a single click through the logs.

Arrange to have SPWrapper find or make up a "tag" and append it to the arguments for the proc. Have each proc accept a final "tag" argument.

Create two new procs: spLog and spLogPop. The former takes the tag and a message and inserts it to the temp table. The latter takes a tag and gives you all the messages associated with that tag, in order, then removes them from the log table. These can be written out to the log file. SPWrapper will execute an spLogPop immediately after any Procedure call.

Lastly, let's cover how you load your procs. As with most things around here, it will be with a cunning script...

#!/bin/bash
db=$1; pass=$2
for x in $(ls ./sql/procs/*.sql); do
  cat "./sql/procs/$x" | mysql -u root -p$pass $db
done;

You'll want to do a bit more in this script, such as checking if password is on commandline, if not to prompt for it (one way suits an insecure dev server just fine, the other is preferable for a production server)

Dedicate a different .sql file to a different task or part of the database or part of your API. Keep things separate and well named. Then in each file, follow this pattern:

drop procedure if exists spSomeFabProc //
create procedure spSomeFabProc

This means all your procs can live with the rest of your app in lovely Source Control heaven and be loaded in to your test database at any time.

Hope this has been useful; Enjoy.

...Click for More

http://www.youtube.com/watch?v=aaSC7qVRL0w

...Click for More

Funny

Real Actors Read Yelp

Video

YouTube

Taming big music collections with MPD and Plait

I may have mentioned before: I'm not very good at listening to music. Or rather, I'm not very good at choosing what to put on.

This is partly due to having a large collection and mainly due to indecisiveness. It is compounded quite heavily by my media set-up, which doesn't make life easy for me to browse and choose.

Today, we'll have a look at solving this heinous First World Problem by tackling it with laziness. Or rather, the pursuit of laziness.

The ultimate aim is to be able to choose music to listen to from my phone with as little effort as possible.

My music is played from my little headless media server connected to an amp. It uses a lovely little program called MPD - Music Player Daemon - which is designed perfectly for use on headless computers. It can be remote-controlled from anywhere, and I use a little phone app to do all the usual things, including building playlists. This bit is a bit cumbersome though.

You may have come across a nice little music player before called "Plait". It is a very simple contraption which, when given a search-word, creates a random playlist including all tracks which mention that word anywhere in their name or path. So, you can search by Album, Artist, Track etc - if you've got your music collection properly sorted. For sorting it nicely and almost automatically, I would recommend "MusicBrainz Picard". But that's a whole other topic.

So today, we're going to integrate the ideas of Plait in to MPD - so that from my phone, I can create a "random" playlist which MPD will let me listen to.

MPD playlists...

First off, we need to know where and how MPD stores its playlists - what format are they in? Quite simply, they are in /var/lib/mpd/playlists and are m3u format. This is nice, because m3u is pretty flexible and forgiving, and lets you simply pop file paths in, one line at a time. The only caveat is that the paths must be relative to your configured "music root" in MPD. Mine is /mnt/bulk/media/audio.

Let's make an index of all the music and try to build an MPD playlist.

For simplicity, let's put the index with mpd. Later on, we'll worry about a better place and file permissions as we build this in to something useable. For now:

find /mnt/bulk/media/audio/Albums -type f -name "*.mp3" -or -name "*.ogg" -or -name "*.flac" > /var/lib/mpd/musicIndex.m3u

Now, let's create a random playlist from a magic word and see if it plays:

grep -i vibert /var/lib/mpd/musicIndex.m3u | sort -R | head -n 50 > /var/lib/mpd/playlists/random.m3u

This will find all my Vibert tracks, give them a quick shuffle and pop the first 50 in to our playlist. Now in your MPD client, simply select the "random" playlist in the list of "server playlists".

So far so good - now we know it is do-able. Time to automate.

Re-indexing on demand...

You may, quite frequently, add new music to your collection, so musicIndex.m3u will need rebuilding. But you only want to rebuild it when it changes, not on some kind of schedule. For this, there's a very useful tool we can use called "incron".

Incron is designed to be a little like "cron", except instead of running a task at a certain time, it runs it when certain filesystem events occur. Install it from the "incron" debian package and have a look at the manual page:

apt-get install incron
man 5 incrontab

Let's configure an incron watcher. As root, edit /etc/incron.d/mpdReindex, put in this line and save:

/mnt/bulk/media/audio/Albums IN_CREATE /usr/local/bin/mpdReindex

This creates a watcher for files and directories being created against the music directory, and will run our reindexer script for each change. Let's have a look at the script:


#!/bin/bash
set -e

# Reindex mpd/plait 
pid=$$
kill $(pgrep mpdReindex | grep -v $pid) || /bin/true
if [ -z "$1" ]; then
 sleep 60
fi

find /mnt/bulk/media/audio -type f -name "*.mp3" -or -name "*.ogg" -or -name "*.flac" -or -name "*.m4a" > /var/lib/mpd/musicIndex.m3u

# Tell MPD to update its DB
mpc update

This is simply the same "find" command to build the playlist as above, but with a small hack: it will first kill any other processes of the same name, excluding our own PID. Then it waits for 60 seconds. This means that it should run a single time just after the last file / directory is created. It's a bit hacky and not 100% guaranteed to work if your audio files are really huge and take over a minute each to copy, but it will suffice for now.

What about triggering the actual Plait-style playlist generation? How about a nice little web page?

The Frontend...

<?php
$words = trim(isset($_POST['words'])?$_POST['words']:'');
$rand = trim(isset($_POST['rand'])?$_POST['rand']:'N');

if( $words == '' )
{
    $x = explode(" ", file_get_contents("./playword"));
    $rand = array_shift($x);
    $words = implode(" ", $x);
}

$words = preg_replace("/[^A-Za-z0-9 ]/", "", $words);
$rand = preg_replace("/[^RN]/", "", $rand);
$chk=$rand=='R'?'checked="CHECKED"':'';

$h = <<<EOT
<html>
<head>
<meta name="viewport" content="width=device-width" />
<style type='text/css'>
<!--
input {display: block; width: 80%; margin: 0 auto 5px auto; border:1px solid #888888; color: #888888; padding: 4px;}
body {padding-top: 10px;background: #e8ffe0;
      font-family: helvetica;font-size:12pt;}
label {display: block; margin: 0 auto 0 auto; text-align: center;}
label>input {display: inline; width: auto;vertical-align: middle;}
input[type='submit'] {color: #444444; margin-top: 10px;}
-->
</style>
</head>
<body>
<form method='POST' action=''>
<input type='text' name='words' value='{$words}'/>
<label for='rand'>Randomise playlist?
 <input type='checkbox' name='rand' value='R' {$chk}/>
</label>
<input type='submit' name='doit' value='Go for it'/>
</body>
</html>
EOT;

if( isset($_POST['doit']) )
    file_put_contents("./playword", "{$rand} {$words}");

die($h);
?>

In the same directory, do this:

touch playword; chmod 777 playword

So now from your mobile, you can simply tap in a new word. But how to build the playlist when this happens? That's right, more incron magic:


/home/web/playword IN_CLOSE_WRITE /usr/local/bin/mpdNewPlaylist

So when the word file is written and closed, a script runs to do something about it. It does it as root*, which lets us do what we need to do to the files, but securely because it is decoupled from the webserver. Notice in the php script above, the preg_replace - that has sanitised the inputs to be only letters, numbers and spaces, so our root-run script should be safe from injection attacks.

* All the script run through incron run as root. If you are implementing this yourself, I would suggest creating a new user with no shell access and using "sudo" in the incrontabs to run the scripts as this user. You'll also need to set permissions appropriately on the files the scripts need to touch. In the interest of clarity, I have left this out of this howto.

Playlist creation...

Now for one last script to create the playlist... /usr/local/bin/mpdNewPlaylist. This is quite a simple one: It is pretty much just the grep from earlier, with some sugar-coating:


#!/bin/bash
# Build the new playlist - randomise if requested
f=$(cat /home/web/playword)
rand=$(echo $f|cut -d' ' -f1);
word=$(echo $f | cut -d' ' -f2- | sed -r 's/[^A-Za-z0-9]//');
out="/var/lib/mpd/playlists/random.m3u"
idx="/var/lib/mpd/musicIndex.m3u"
opts='';
if [ "$rand" = 'R' ]; then
 opts="-R"
fi
grep -i "$word" "$idx" | sort $opts > "$out"

# Load it in and gogogo!
mpc clear
mpc load random 
mpc play

And the cherry on the cake - use mpc to clear the current playlist, load the new one and set it going!

Enjoy.

...Click for More

Article

incron

MPD

Music

Plait

http://www.youtube.com/watch?v=TcvZL6fx-lU

...Click for More

Chemistry

Dubstep

Music

Video

YouTube

Stored Procedures for fun and profit

I'll let you in to a little secret: I love Stored Procedures. Don't tell anyone though, will you, I wouldn't want that getting out.

So much of what we do these days deals with data, and lots of it. And often, that means relational databases. A lot of people like to drop simply SQL queries directly in to the [PHP, Java etc etc] code they're writing and then further manipulate the data that comes out in [PHP, Java etc etc] code, but that is ignoring all that a decent DBMS can do: In a Stored Procedure you can guard all that data properly, access it securely in a handful of clearly defined ways and manipulate it in so many clever ways before delivering just the results that are needed to your app.

So what makes them so incredibly useful?

First off, they are secure. Sure, you can do insecure things with Stored Procs if you really try, but generally they are easier to secure than ad-hoc queries. This security comes from two features:

The variables passed in are strongly typed. One can only pass an int in to an int argument, string in to a varchar, etc. MSSQL will complain bitterly if you get this wrong, and MySQL will implicitly cast to the right type. Unless you are doing something with Dynamic SQL within the procedure, the arguments are passed in to any internal queries by value, so no worries with SQL injection here.
If every data manipulation your program performs is via a Stored Procedure, then the user with which your main program (The PHP, Java, Python, etc etc) connects to your database needs only have "execute" access. So any other kinds of SQL injection which might attempt to select, update or drop tables simply will not work.

They are fast to execute. This is actually quite a contentious issue, as it is implementation-specific. But the theory is that all the work the query compiler and query analyser do to figure out how best to process the SQL you've written only needs to be done the first time a procedure is run. This is only saving milliseconds, but it could make quite a difference on a very loaded web server running the same SQL many times a second.

They are fast at dealing with large data sets. This is what a DBMS is for: crunching large amounts of data. If you extract large amounts of data and try to crunch it in an interpreted language, it will be slow. A Stored Procedure allows you to crunch and re-crunch data, passing it through many stages, until just the small amount of data you wish to render to the user is returned.

They let you test your data manipulations away from the web. This is a specific bonus in my area of programming. If you can black-box your data manipulation and retrieval in to layers of Stored Procedures, you can test the data input and output before you write the PHP that renders it. Once you get in to the habit of encapsulating your data in this way, it becomes much easier to later manipulate the data logic without breaking the rendering of that data: you have completely split data manipulation from user interface rendering.

They let you validate close to the inserts and updates, easily doing complex multiple inserts across many tables quickly and neatly with the minimum of input data, returning user defined errors easily if that input data is somehow wrong or if the user does not have permission to perform a particular operation.

They help you maintain integrity by providing well-defined ways of accessing and updating your database.

They let you create complex models of your data. An SQL server is not just a program that orders and contains your data, it is also an advanced execution and data mining environment. It is specially designed to let you crunch big data, and there is very little that isn't possible in data manipulation that can't be done within the confines of a Stored Procedure. Performing all calculations, manipulations and condensing of your data before it leaves the server is very fast, and leaves your main program free to simply render the results on screen. You are almost never showing the user a large results set on screen, so why should you have your DBMS transfer a large results set back to your UI code?

By using different complex models of your data, you get an added benefit of being able to very easily create different "renderers". By segregating all data processing to the database, you can use this single, rich, complex datasource to fuel a PC layout, a mobile layout, CSV/report generation and JSON / XML web-services using very generic code and no extra, costly transformations.

If this has made you curious enough to see what you can do with Stored Procedures, I'd recommend chapters 13 and 19 of the MySQL Reference Guide: http://dev.mysql.com/doc/refman/5.1/en/index.html

In future articles, I'll show you how to fully integrate Stored Procedures in to PHP, allowing you to create an easily callable database API, complete with logging, exception handling and low level authentication.

Enjoy.

...Click for More

Article

Programming

SQL

Stored Procedures

http://www.youtube.com/watch?v=wRIVZjfmr2A

...Click for More

Chill

Deep Dive Corp

Music

Video

Wider

YouTube

Puns and Street Signs

http://www.thepoke.co.uk/wp-content/uploads/2012/08/funny-street-signs-17.jpeg

...Click for More

Funny

Image

Jpeg

Tearable puns

http://www.youtube.com/watch?v=VqQhkX5stJU

...Click for More

Breakbeat

Funky

King of the Bongo

Music

Video

YouTube

Bingo

I'm fairly allergic to corporate management speak. If you too don't like people constantly trying to run their hymn-sheet up your flag pole, try handing out some of these in the next meeting you get diarised to action -

http://www.bollocksphere.co.uk/Content/Business.shtml

...Click for More

Bingo

Business Speak

Funny

Management Speak

There's something awfully strange about this lady and her apple obsession...

http://iloveyoulikeafatladylovesapples.com/

...Click for More

https://soundcloud.com/dj-nee/dj-nee-like-a-drug

...Click for More

Ambient

Audio

DJ Nee

Hardcore

Like A Drug

SoundCloud

https://soundcloud.com/autobots/the-autobots-ft-roisin-brophy

...Click for More

Audio

Autobots

DnB

Music

SoundCloud

oEmbed for external media

Have you ever struggled to embed external media in a web page? So many providers to choose from...youtube, vimeo, soundcloud, mixcloud, just to name a very few, and they all have completely different URL structures, different players, different...everything.

The aim of the game is to be able to take, say, a standard SoundCloud URL and paste it in to your page, and have the page somehow turn it in to embedded media.

Step forward oEmbed. This is a new, informal "standard" starting to be adopted by a lot of providers.

The theory is simple - the provider creates an oEmbed webservice, eg http://soundcloud.com/oembed, which when tickled with a request containing a standard URL of that provider, returns a JSON object containing much well-defined meta-data about the media in question.

Lots of info on the format of the request you make and the data returned can be found at http://oembed.com/

The bit you need to know for embedding is that the "html" property in the response will contain the provider's embed code for this media. Append this to your document somewhere and you're done.

And a provider is expected to support JSONP (Otherwise it wouldn't work cross-domain).

Enjoy.

...Click for More

Article

Audio

embed external media

JSONP

mixcloud

oEmbed

SoundCloud

vimeo

Oldskool rave, Dreamscape style.

http://soundcloud.com/dreamscapeevents/dj-dougal-dreamscape-19th

...Click for More

Audio

DJ Dougal

Dreamscape

Mix

Music

Oldskool

Rave

SoundCloud

Picture Dump 1.

No urls found!

...Click for More

http://www.youtube.com/watch?v=W8Y3piodWes

...Click for More

DnB

Music

Red Hot Car

Squarepusher

Video

YouTube

Ben Goldacre's new book "Bad Pharma: How drug companies mislead doctors and harm patients"

(The sequel to "Bad Science", which will be on every school's reading list when I run the world)

http://www.amazon.co.uk/exec/obidos/ASIN/0007350740/b3ta-21

...Click for More

Bad Pharma

Bad Science

Ben Goldacre

http://www.youtube.com/watch?v=7p3tmuPqe54

...Click for More

Dark

Electronic

Music

Radioactive Man

Trespasser

Video

YouTube

The Two Ronnies on Mastermind...

http://www.youtube.com/watch?v=y0C59pI_ypQ

...Click for More

Funny

Mastermind

Two Ronnies

Video

YouTube

https://soundcloud.com/djjuxtapose/9pm

...Click for More

ATB vs Juxtapose

Audio

Breakbeat

Music

SoundCloud

http://www.youtube.com/watch?v=RPVAipmV7jY

...Click for More

Badly Drawn Boy

Indie

Music

Once Around The Block

Video

YouTube

How To Green The Deserts

http://www.youtube.com/watch?v=vpTHi7O66pI

Without wishing to spoil the ending, this chap has some very simple ideas on how to turn the world's deserts back in to habitable land, with water and much grazing for animals. And not just ideas - he has spent 20 years putting these ideas in to practice, and large swathes of Africa are now no longer starving because of this chap.

We're lost a startling amount of land over the last few hundred years to deserts due to our own impacts on the world. The key to greening these areas again is treating the land as it once was by the huge herds of herbivores.

The results are absolutely incredible.

...Click for More

Allan Savory

Article

Green Deserts

Reverse Climate Change

Video

YouTube

Safe Nuclear Molten Salt Reactor

This makes me happy. Those clever people at M.I.T have designed / improved on designs for a kind of Nuclear Molten Salt reactor which allows us to use the 97% of fuel which is left in each Uranium rod, without reprocessing. "Simply" dissolve the spent Uranium in molten salt and feed this special reactor with it and almost all of what is left can be used.

We have enough waste Uranium lying around to power the whole world for 80 years. And we have millions of years worth of Thorium.

See, the problem we face at the moment is that we're too fixated on global warming. It is the ultimate "Will-they-won't-they" soap opera. The elephant in the room is that the oil and gas will likely run out first, and unless the equatorial countries can be persuaded to set up massive solar farms to power the world (Which the Americans wouldn't allow), we are essentially screwed.

Welcome to the cure:

http://www.theregister.co.uk/2013/03/14/nuclear_reactor_salt/

...Click for More

Article

Nuclear Energy

Safe

Salt Reactor

Happy 100th episode, Robot Chicken.

http://www.youtube.com/watch?v=PzzqE7EPoIc

...Click for More

100th Episode

Cartoon

Robot Chicken

The Rescue

Video

YouTube

Fast wildcard searching in SQL

Here's a trick I read about some years ago, I've only just had a chance to use to full, brutal effect.

Did you know: If you use "like" in an SQL statement and put '%' on the left side of your search term, you no longer use indexes?

Your SQL Server then has to examine every single row of the table you are searching. If you have more than a few thousand rows, this is painful.

The reason is that the wildcard on the left hand side of your search term stops your term fitting nicely against the left hand edge of the index. Think of it like looking up "John%" in the phone book - you can jump straight to the "J"s, then "Jo", and so on. Once you've found John, you only have to scan a few pages to find all the "John%" - the names starting with "John". Nice and fast. As soon as you search instead for "%John", you no longer have any insight in to where to look: you have to look at every page of the phone book. .

This is pretty slow.

If you are willing and able to sacrifice some disk and memory space, here's how you fix it and get to always do lightning fast index-based searches:

Duplicate your search column and place in it the same data, but backwards. Add an Index for both searchCol and searchColReverse.

Then to search, INNER join your search table twice, searching tab1.searchCol with John%, and tab2.searchColReverse with nhoJ%.


select foo.*
from foo
  inner join names as tab1 on tab1.fooId = foo.Id
  inner join names as tab2 on tab2.fooId = foo.Id
where 
  tab1.searchCol like 'John%' 
  and tab2.searchColReverse like 'nhoJ%';

Each of these joins will use appropriate indexes, and the small results sets will be merged, removing results which do not appear in both sets.

Enjoy.

...Click for More

Article

Fast wildcard Searching

Programming

SQL

Tip

https://www.youtube.com/watch?v=YIH0VJGB6Es

...Click for More

DnB

High Contrast

Music

Scribble

Underworld

Video

YouTube

https://www.youtube.com/watch?v=VyqqBpT7XWU

...Click for More

Atmospheric

Chill

Dickinsons Rocket

DnB

Komatic & Technicolour

Music

Video

YouTube

Boxcat 1, stairs 0. Or is that the other way around?

https://i.chzbgr.com/maxW500/7022896640/hE0A13C4C/

...Click for More

Animated

Boxcat

fun

Gif

Image

Stairs

http://www.youtube.com/watch?v=7p3tmuPqe54

...Click for More

Dark

Music

Radioactive Man

Video

YouTube

http://www.youtube.com/watch?v=XjYGpTkoRVw

...Click for More

Blues Traveller

But Anyway

Funky

Indie

Video

YouTube

http://i.imgur.com/ExUJS.jpg

...Click for More

Funny

Girraffic Park

Image

Jpeg

http://www.youtube.com/watch?v=sP4NMoJcFd4

...Click for More

Can't hug every cat

crazy cat girl

eharmony

Funny

Video

YouTube

http://www.youtube.com/watch?v=QJ4Pm0N8s78

...Click for More

Air

Alone in Kyoto

Chill

Music

Video

YouTube

The whole original Hitch Hiker's Guide to the Galaxy TV series... on a YouTube playlist. Nice.

http://www.youtube.com/watch?v=uLwreD9NL7c

...Click for More

Hitchhiker's Guide to the Galaxy

TV Series

Video

YouTube

http://www.youtube.com/watch?v=j03XaHpl6g8

...Click for More

Funny

Video

What a Complete and Utter Countryside

YouTube

http://www.youtube.com/watch?v=gvLoVCIhmyE

...Click for More

Amo Bishop Roden

Boards of Canada

Music

Video

YouTube

http://vimeo.com/39507968

...Click for More

Alien Adventure

Funny

List of crazy patents... http://www.freepatentsonline.com/crazy.html

My favourite, number 2: "Apparatus for facilitating the birth of a child by centrifugal force"

...Click for More

Crazy Patents

Funny

http://www.youtube.com/watch?v=Vnp5NvKQwUU

...Click for More

Chill

Music

Sound in a Dark Room

Telefon Tel Aviv

Video

YouTube

http://www.youtube.com/watch?v=vMmqgBvNdVQ

...Click for More

Always

Chill

Jazzy

Music

Submotion Orchestra

Video

YouTube

http://www.youtube.com/watch?v=EvvjgDIC3e4

...Click for More

Funny

Grasshopper Mouse

Video

YouTube

http://www.youtube.com/watch?v=s-rUPIzgTGI

...Click for More

Indie

Matchbox 20

Music

Unwell

Video

YouTube

https://soundcloud.com/christian_hoffmann/toronto-is-broken-spirit-song

...Click for More

Audio

DnB

Music

SoundCloud

Spirit Song

Toronto is Broken

http://www.youtube.com/watch?v=G0smSlYifgE

...Click for More

Dear Joan

Indie

Matchbox 20

Music

Video

YouTube

http://www.youtube.com/watch?v=Aw8i28bNoYY

...Click for More

Air

All I Need

Chill

Music

Video

YouTube

http://www.youtube.com/watch?v=EmStlUk8dKE

...Click for More

Classical

Le Onde

Ludovico Einaudi

Music

Video

YouTube

https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash3/575331_476931739002951_151567507_n.jpg

...Click for More

Cats

fun

Image

Jpeg

http://www.youtube.com/watch?v=TdxNGykpkrY

...Click for More

Boards of Canada

Chill

Music

Peacock Tail

Video

YouTube

http://www.youtube.com/watch?v=ACqVAfMzngs

...Click for More

Bagleys

Funny

Raveageddon

Video

YouTube

https://fbcdn-sphotos-c-a.akamaihd.net/hphotos-ak-snc6/255210_10150979565273595_937455482_n.jpg

...Click for More

Cats

Funny

Image

Jpeg

http://www.youtube.com/watch?v=XOurjE9s3GA

...Click for More

Bacon

Bacon wrapped eggs

Food

Video

YouTube

http://www.youtube.com/watch?v=yXRWRvRxgvg

...Click for More

DnB

Music

Reach

Tantrum Desire

Video

YouTube

http://www.youtube.com/watch?v=eeSyHgO5fmQ

...Click for More

Aqua Death Lens

Lens

Science

Video

Water

YouTube

https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-ash4/431381_468927876458865_1113702809_n.jpg

...Click for More

Image

Jpeg

Malcolm X

Newspapers

Oppression

http://www.youtube.com/watch?v=XRj91VZG4Dw

...Click for More

Deadmou5 & Kaskade

DnB

I remember

Video

YouTube

https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/311210_10151095942378017_1516965470_n.jpg

...Click for More

Funny

Good Dog

Image

Jpeg

http://www.youtube.com/watch?v=1_oa8m5Oq00

...Click for More

Magic

Video

YouTube

http://www.youtube.com/watch?v=SIaFtAKnqBU

...Click for More

Funny

Screaming sheep

Video

YouTube

https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/392159_10151122276689719_137745234_n.jpg

...Click for More

Funny

Image

Jpeg

Statue beating

http://www.youtube.com/watch?v=cMa9QNlUglM

...Click for More

Ambient

Chill

Music

Native Force

Video

YouTube

http://www.youtube.com/watch?v=yx-Qhh3EjyM

...Click for More

Atmospheric

DnB

Hardstep

Music

Video

YouTube

The rules of deployment

FileNotFound: /home/vhosts/olicomber.co.uk/web/downloads/softdev/installer FileNotFound: /home/vhosts/olicomber.co.uk/web/downloads/softdev/loadAllSql.sh FileNotFound: /home/vhosts/olicomber.co.uk/web/downloads/softdev/copyDB

That wonderful software product you've developed and want to put live, how do you know it is going to work?

You have arranged with your client that their new software will be put live, today, out of hours. It's been a long day, you're tired, if anything goes wrong, you know you're past fixing it. Nervous?

Why?

You tested it, it's going to work, so you start going through your deployment process...it can't fail, right?

I met a friend for lunch some time ago. He was most amused telling me the story of his morning. His colleague was going on holiday, but the project he had been working on for a client was due to go live whilst he was away. It was Very Important Stuff, though clearly needed signing off before going live, so this wasn't going to happen today.

His colleague had handed him a handwritten sheet of paper, listing all the files which needed to be copied from an export of HEAD in CVS over the top of their counterparts on the live server. Alongside this were other instructions on directories to create, config files to change and .php files to manually adjust on the live server. This was going to be very slow, nerve-racking, and would leave the system slightly broken for short amounts of time however fast he typed.

We chuckled about this for a while in disbelief, speculating about how much effort it might be to restore this server to its current state if it blew up, given all the random files being copied around manually in the name of deployment. Luckily, nothing came of it as the manager involved forgot to sign it off for a couple of weeks. So a happy ending, of sorts, at least for my friend.

But if you don't notice anything wrong with this situation, if you think the stress and the time taken to do a deployment like this is just a part of the fun of it, you should read on.

So what's the key to successful software deployment? Everything. Everything done on this project since its inception leads up to its perceived success. Unlike all good stories, let's start at the end.

Note: This is written from a "PHP running on a LAMP stack" perspective, but the principals should translate well to almost any other set-up.

Pressing the button

What does it mean, to "press the button"? To me, and I am very lazy, it means to press as few buttons as possible to deploy software. No complication, no stress, knowing it will just work. Unlike in the story above, I want to have some kind of program do everything for me - to remove the old code and put the new code in place, to load stored procedures in to the database, to do any other housekeeping.

Write an installer. It doesn't have to do anything clever, and in fact you should write all your code in such a way that nothing clever needs to happen at install time:

Where configuration options are not in the main config file, have sensible defaults used or warnings sent out.
Make it so that no code needs to be changed in the production system.
Make it so the whole working directory (vhost) can be deleted and replaced without any other work needing to be done: Place your site configuration file outside the vhost. Make sure any downloadables which are not in the database are outside the vhost. Make sure temporary files/caches are not relied on, such that they are outside the vhost.

I can't stress that enough: No code should ever be manually changed on the production system. There just isn't any need for it, even for reproducing bugs that only seem to happen on production data (see below). You run the risk of introducing bugs now or having your changes removed by the next proper deployment.

To be technical about it, a layout like this works quite nicely to fulfil all these requirements:

Config file:
    /home/web/olicomber.co.uk/web.conf.php
Vhost directory:
   /home/web/olicomber.co.uk/web/

The code inside the vhost can easily find the config file as it is named after the directory it is in and is one level up.

Now, you might baulk at the idea of writing an installer. Your code needs to be installed and live nownownow, you don't have time to write an installer. So don't. Write an installer creator, which will take your lovely, well tested code and wrap it up in to a single file with an install script on the front. There is an example one attached at the top; feel free to modify it and use it. Protip: Where it is being modified to use in a single project, change the program arguments such that only the bare minimum needs to be typed each time. For instance, your Install Paths are always going to be the same, as is your project name. In your case, perhaps only the "release tag" is needed - be lazy, minimise your typing, minimise the knowledge you might need one day to hand over to a friend or colleague, minimise potential mistakes.

What else might we need to do when we "hit the button"? With the new code in place, it should be a matter of upgrading the database and the config file. And that is only if this is a "major" release.

Maintain upgrade.sql and readme files

Inside your code directory, keep a /docs. Use htaccess to deny any access and file away any information on APIs, error codes, an upgrade.sql file and a readme file.

In your upgrade.sql file goes any database alter command or any data loading. If you are using phpMyAdmin, it will give you the SQL to pop in here when you run it on your dev database. If writing by hand, obviously just copy and paste it from your console.

To go live, simply:

cat upgrade.sql | mysql -u root -p myproddb

In your readme file goes anything else you might need to remember for later - mostly setting things in your config file, but maybe supporting packages to install in production too. This is your checklist. Get in to the habit of checking this before deployment and write it in a way that means you can just copy and paste to production any commands you might need to run. The aim of the game is to minimise mistakes. The sooner you're finished, the sooner you get dinner.

One could automate these things, but personally I prefer to have these steps under manual control.

But how do you know that when you're "pressed the button" it will all work?

Test your installer, test your code, test your DB

For the deployment to be immediately successful, you need to have well tested code and scripts.

But what does it mean to have well tested code? You can just check things seem ok in your dev area and then push straight to live, right?

What we are after here is confidence. Confidence that our software will work properly when put live, that it will behave properly with live data, that the live database can be upgraded cleanly and perfectly by your script.

Three steps to deployment testing heaven:

Branch your code to a new area with a separate configuration file, configured for a separate database.
Create and maintain a "Live Mirror" database
Copy your Live Mirror to your testing database and run your upgrade.sql script over it

Step one should be fairly self explanatory: You should not test in your dev area. You should create a separate area for testing, from scratch. Remember: be lazy. This should be fast to do, really just a single local branch of your code. More on branching later.

Step two: a live mirror. You don't want production data on your development server, it probably isn't secure. But you do want a database which has that volume of data, where that data is completely natural and follows certain patterns.

The way around this paradox is with anonymisation. And another script. You want making a new Live Mirror to be this easy:

Production server:
  mysqldump -u root -p livedbname | gzip -c9 > livedb.sql.gz
Dev server:
  scp prodserver:livedb.sql.gz .
  buildLiveMirror ~/livedb.sql.gz
Production server:
  rm livedb.sql.gz

When you get back from making a cup of tea, your new Live Mirror database will be created. But what's that script do?

Drop and create myproj_livemirror database
Load in that .sql dump file
Run an anonymising .sql script over that database
Remove that dump file of live data

Step three is where the magic happens: For every table which contains sensitive data, such as name and address, usernames, (hashed - always hashed) passwords, dates of birth, credit card numbers etc, you will have a line in that script to do an "update table" and replace each letter or digit in that field with a random character of the same type (letter or number). You can write an SQL function quite easily that will do this, taking the current field contents and returning new field contents. Do not use cryptography here, we are looking to replace the data with something that looks similar but is utter garbage. Cryptography would slow the process down and potentially make it insecure by making it reversible. It does not need to be reversible: In a customer name, it doesn't matter what a letter is, as long as it is upper or lower case and the name is the correct length. Same with a credit card or phone number. Preserve non-alphanumeric characters, such as spaces or apostrophes.

As a nice-to-have, have the script change the admin password to something easy for development. There is nothing of interest now in your database, so just "password" would do nicely.

This anonymisation is obviously a slow step, that's why you now have a nice hot cup of tea in your hand.

Note: I have not included a script here for example, as by this point you should be able to happily create a script to drop/create a DB, load in a dump file and execute an SQL script on your database. Your anonymisation script is personal to you.

What are you going to do with this nice, accurate, live mirror of your production database? Nothing. Absolutely nothing. You're not going to touch it. You're going to copy it. This bit is a lot faster.

./copyDB myproj_test

Another time saving script you will use many, many times: This will drop the database you specify, make a new one and copy your myproj_livemirror to it before running another script to load your stored procedures (more on this later).

Now you have a nice, fresh database to test with, you can test your upgrade.sql script.

cat upgrade.sql | mysql -u root -p myproj_test

If it works, great. If not, fix and repeat the copyDB step to copy the _livemirror to a new _test DB and try again. Once all good, go through your app - at least the bits you know the spec applies to at least - and make sure it works.

At this stage, a full regression test of your product might be in order. You could even go through the last few steps to create a Staging area, completely separate from Dev, Test and Production, where your colleagues and even your client can have a play safely.

You're developing on a production quality database, testing code for deployment, but how do you know it will work on your production server?

Production Environment = Development Environment

Of course, all of this assumes that your development and production servers are in sync version-wise. I like to do this in several ways:

Use a Linux server distribution that has major version numbers.
Do identical installs on Production and Development - say, Ubuntu 12.04 minimal, with no extras.
Use a single "apt-get install" command to install your LAMP stack and all the PHP modules you think might be required.
Put these few details in a page on your Wiki. If you don't have a Wiki, set one up, they are invaluable for specs, notes, howtos, asset registers and, as here, server build details
Make sure the systems are regularly dist-upgraded. If you install a package on one that relates to your code (ie, a PHP module), install it on the other. Note this in your readme file / production checklist.

Simple, huh? It should be. If you think this advice is too obvious, think again about the environments you have coded in, on ancient servers, which in no way match development, where the sysadmins are scared of the day they might have to build a working replica of one from scratch.

Here's a fun thing you might like to do on your production and development servers: Use a Source Control tool to version control your configuration files. More on Source Control later, but for now it is enough to know that you can use a tool like bzr to keep old versions of every configuration file.

cd /etc
bzr init
bzr add *
bzr commit -m "initial"

The set of commands above will place every file under /etc under control of bzr. When you change any Apache config or vsftp or exim or any number of other programs' config files in /etc, if you are happy with the changes you can "bzr commit" to tell bzr to take these files as the next version to store. If you later change something and are not happy with the changes, you can diff to see what changed or you can revert to undo all changes. Everything stays sane, everything stays happy, you stay lazy.

So now you know your development and production environments are in sync, your database upgrade will work, your code is tested and works and you're ready to press the button.

But meanwhile, back at the range... where did you create the installer from? How did you write this code? How have you kept this code in any way sane over the many thousands of lines you've written in an abstract order?

Source Control for fun and profit

What is "Source Control"? It is a way of managing how your code changes over time. You may have heard of bzr, git, mercurial, cvs... these are tools that help you incrementally change your code, review changes and, eventually, export a working version to test and eventually install to Live.

Good use of good Source Control software can save your skin.

I prefer bzr, so any examples will be based on bzr commands. But most tools like this have many common operations, just differing syntax.

If you want to learn bzr, git, mercurial, etc, there are plenty of resources online which will explain these commands and all the theory behind. Here, we will talk about how to use these tools effectively to help you in the deployment process.

The key to making sure of a good deployment is making sure your "trunk" is always, within reason, stable.

What do we mean by trunk?

You are maintaining a Source Code "tree", which has a trunk. At the start of the project, there is only a Trunk. You add files to the project, edit them and "commit" them. To "commit" means to push them in to Source Control, saving new versions of all changed files on to Trunk. You have made Trunk longer. The previous versions can all be pulled out, and one can compare the differences between any two versions of a file any time.

Tree:

--->(commit v1)--->(commit v2)--->(commit v3)--->0

"0" is the tip of the "Trunk" branch. You should aim to keep this as close as possible to Live. If you have a bug on Live, you fix it on Trunk, test and promote Trunk to Live as quickly as possible. If something goes wrong with the new release, you export the previous version of Trunk and put that Live.

Your client will approach you at some point and ask for a new feature. This feature might take a few hours, days, weeks or even months. How can you develop this feature but keep Trunk identical to Live? You create a Feature Branch.

A Feature Branch (Or just Branch) is a snapshot of the code you can develop in parallel to Trunk. This lets you keep Trunk clean but carry on development. When you are happy, you Merge the Branch on to Trunk, do some testing and promote to Live.

Here's how your Source Tree will look with a Branch:

--->(commit v1)--->(commit v2)--->(commit v3 - bugfix)--->(commit v4 - bugfix)--->0
                                    \
                                      \----->(commit v2.1)---->(commit v2.2)---->1

So, at the point where the code was "v2", we created a Branch. The code was changed and committed several times, independently of the Trunk. At the same time, bugs were found on live, fixed on Trunk and Trunk was placed live - completely independently of your new whizzbang feature.

What bzr will do for you is fold or merge "1" on to "0". It will take all those changes you made in all those files and give you a new version which combines all those changes, quickly and sanely.

Now you can test, build your installer and deploy from Trunk. If you really want to be crafty, and you know testing will take some time (days, weeks), you can make a new testing branch and merge everything on to there and test. This way, your trunk stays clean and in step with Live, allowing for bug fixing.

The two other most useful functions of bzr are tag and export. You may have wondered earlier what a "Release Tag" was...read on.

When you want batch up all your files and stamp them as "this is all ready to test or go live", you tag them with something descriptive. You can then always refer to this exact set of files at this exact point in time when you export or when you need to check for differences between versions.

So you might tag your testing branch as release_1.30_RC1, then push to your testing/staging area. As bugs are found, you rev this to RC2, RC3 etc. When you are ready for production, you finally tag as release_1.30 and instruct your installer creator to build you an installer for release_1.30.

Now you can export, test and deploy, nice and clean.

But how do you know what you have written up to now is what is needed?

Spec small, spec clean

Well, this is the million pound question. How do we write software that people want to use? How do we satisfy our clients? You may recognise some of the below, and indeed the above, from the Agile or Extreme programming methodologies. I see a lot of these methodologies as common sense or obvious, and the idea that these rules are collected together in one place and called a methodology is a great way to give the business confidence that much method is being applied to the madness, and that software quality is being maintained.

Never write a whole spec for the product in one go. You are going to be working on this product for a long time and much will change.

By all means, sketch out what the interface will look like overall, where things might go eventually. But not everything has to be planned in minute details from the beginning.

I'm not saying do not plan; quite the opposite. Plan, but only plan for what is needed now, but with some thought as to what will be implemented next. Do not plan and write code "just in case", write it with a mind for what may be required later, but plan to change it later: refactor regularly. Don't write generic code when you have a single use-case.

From a design point of view, try to picture your product as a set of features which need to be implemented in a certain order, and that your customers will be using each feature as soon as it is ready and stable.

Break each feature down in to logical blocks, plan to do similar features around the same time and, where possible, make them share code.

Do not write generic code unless you have multiple use-cases. Write re-usable code, write specific code. If you find a use-case which is similar in spirit to something you have previously coded, write now and refactor later.

Try to re-use UI features, but do not write main driver functions to control disparate UI features. Your users will thank you for the consistency, as will your software testers.

If you find yourself writing "generic" functions which take huge numbers of arguments to support multiple uses and contain much "spaghetti" code to decide what to do, you are doing it wrong.

If you're working in a UI, look at each screen as a set of features, each of which is a living, breathing entity, and write down how it should behave. A feature could be a single screen widget or it could be a whole process of finding and displaying data. Make it, where possible, so that a feature is self-contained. By that, I mean it should be useful in its own right or add on to something else. "Search" is not a feature unless you also implement "Search Results". Though "Filtering of search results" would be a self-contained, useful add-on. And from this, you have a "feature spec", which leads to an implementation and test plan.

Break down the features to logic blocks until you can picture pieces small enough to code and can attribute time to these problems...a few hours here, a few days there. Note down in your spec any unknowns to fill in the gaps later and also jot down ideas for solving particular problems or useful libraries you think should be used, or design patterns which will make something reusable, because remember, you're not planning anything else right now but you are thinking about everything.

Use bullet-points for requirements as well as a clean description of behaviour. Your client is going to read and understand these, and people are lazy.

Prioritise these features in order of use and put them in phases, where each phase will last for a few weeks at most. At the end of a phase, there may be a fully tested product installed in to production - or there will at the very least be a client demo. This involves your client in the process, which keeps everyone happy.

Always be happy to throw away bad code. Demo early, demo often, let your client lead you. If you are throwing away code and reimplementing it due to refactoring or you (or a colleague) has found a better way, be happy you have learned something.

But never try to implement too much in one go - your end product will have too many features and requirements to describe in a single spec.

Now you have planned, branched, coded, tested, merged, tested, db+install+code tested and deployed a feature, and it was almost entirely stress-free.

But where do these "features" come from?

Talk to your users

They are, after all, the people who will have the dubious pleasure of using your software on a daily basis. Sit with them, watch what they do. If they are describing what they have to do with their current software and are pointing out problems with it, ask yourself "should it be implemented this way without the bugs or has their last programmer approached this problem from completely the wrong angle?".

Think about the way they work and imagine doing their job for them. Are they doing too much? What can be automated?

Make suggestions. Word them as questions. Never, ever be afraid to be wrong, as you will never learn anything if you are always right.

Go away and think. Sketch things. Go back to them with several ideas. Involve your users in the whole process, make them proud of what you create together. One of the benefits of working on the smallest, most useful cogs at a time is that you can always throw it away or change it if it is wrong. Be flexible in what you produce and you'll turn your users in to excited designers.

The beginning.

...Click for More

Article

Deployment

installation

repeatability

software development

source control

Oh my GOSH this guy is amazing.

http://www.youtube.com/watch?v=5vCcZIARw9k

...Click for More

Ain't Nobody

Amazing

guitar

jon gomm

Music

Video

YouTube

Absolutely, totally, utterly useless websites.

http://www.theuselessweb.com/

...Click for More

fun

silly

useless

websites

Code South Catchup

If you've missed any Code South over the last four days, catch up here.

https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_19:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_19:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_18:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_18:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_17:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_17:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_16:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_16:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_15:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_15:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_14:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_14:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_13:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_13:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_12:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_12:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_11:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_11:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_10:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_10:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_09:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_09:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_08:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_08:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_07:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_07:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_06:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_06:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_05:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_05:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_04:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_04:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_03:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_03:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_02:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_02:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_01:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_01:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_00:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_08-12-2021_00:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_23:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_23:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_22:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_22:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_21:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_21:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_20:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_20:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_19:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_19:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_18:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_18:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_17:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_17:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_16:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_16:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_15:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_15:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_14:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_14:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_13:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_13:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_12:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_12:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_11:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_11:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_10:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_10:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_09:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_09:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_08:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_08:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_07:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_07:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_06:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_06:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_05:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_05:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_04:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_04:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_03:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_03:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_02:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_02:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_01:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_01:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_00:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_07-12-2021_00:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_23:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_23:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_22:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_22:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_21:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_21:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_20:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_20:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_19:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_19:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_18:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_18:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_17:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_17:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_16:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_16:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_15:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_15:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_14:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_14:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_13:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_13:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_12:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_12:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_11:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_11:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_10:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_10:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_09:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_09:00.mp3 https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_08:00.oga https://www.olicomber.co.uk/downloads/codesouth/CodeSouth_06-12-2021_08:00.mp3

...Click for More

Article

Code South

Playlist

Auto-mounting USB sticks on a Linux NAS

Wouldn't it be nice if your USB sticks auto-mounted somewhere nice and easy on your Linux home media server, so you could play external media, perhaps controlled via VLC Remote from your phone, as soon as you plugged it in? Here's how you do it.

SSH to your server and become root. Go to /etc/udev/rules.d and create a file 80-usbmount.rules.

Put this in the file:

SUBSYSTEM=="block", ACTION=="add", ENV{ID_USB_DRIVER}=="usb-storage", ENV{DEVTYPE}=="partition", RUN+="/usr/local/sbin/usbPlug 'add' $env{DEVNAME}"
SUBSYSTEM=="block", ACTION=="remove", ENV{ID_USB_DRIVER}=="usb-storage", ENV{DEVTYPE}=="partition", RUN+="/usr/local/sbin/usbPlug 'remove' $env{DEVNAME}"

Restart udev:

service udev restart

So now when a device is inserted in the "block" device subsystem, using the "usb-storage" driver, and it is a "partition", the /usr/local/sbin/usbPlug script is called to work some magic.

You can get a full list of device attributes using the usbadm command:

udevadm info --query=all --name=/dev/sdc1

Any output line starting "E:" is a device property value you can test with "ENV{key}".

It will run the script with "add" or "remove" as a command and the full device path. So, now we need a script which can mount this device in a known, easy location. Save the following as /usr/local/sbin/usbPlug and "chmod 700 /usr/local/sbin/usbPlug" to make it executable:

#!/bin/bash
set -e

# Called by udev when a usb disk is plugged in.
# Tries to mount it in a known place.

args=$*
logger "usbPlug got: $args"

mpath="/mnt/bulk/media/video/"
mbase="usb"

if [ "$1" = "remove" ]; then
 mpoint=$(mount | grep $2 | cut -d' ' -f 3)
 logger "Unmounting $mpoint..."
 umount $mpoint || /bin/true;
 if [ -z "$(ls $mpoint)" ]; then rmdir $mpoint || /bin/true; fi;
 exit 0;
fi

for(( dnum=1; dnum<5; dnum++ )); do
 mpoint="$mpath$mbase$dnum"
 if [ -d $mpoint ]; then
  if [ ! -z $(ls $mpoint) ]; then
    # Dir exists and not empty - ignore
    continue;
  fi
 else
    mkdir $mpoint || /bin/true;
 fi

 # It exists and was empty or was just created...
 logger "Mounting $2 on $mpoint..."
 mount $2 $mpoint || /bin/true;
 break;
done;

So what does this do? If it receives a "remove" command and a device, it finds out the mount path of it, unmounts it and (if empty) removes the directory/mount point.

Likewise, if it receives an "add" command and a device, it tries to create a directory and mount it. The directory will be named usb{1,2,3...}, depending on what is available to use. Its path is in the home of my media, so it is easy to find from the remote.

Enjoy.

...Click for More

Article

Auto Mount

hotplug

Linux

Media Server

NAS

udev

USB

Configuring iptables for FTP

Isn't IPTables great? You read and you read and you tinker and you work it out, and suddenly everything drops in to place and it seems like the most obvious way for a firewall to work.

And then, because it works, you don't touch it for a year...

So here is how to set up for FTP access, both Active and Passive, in three lines:


/sbin/modprobe nf_conntrack_ftp
iptables -A INPUT  -p tcp --dport 21 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp  -m state --state RELATED -j ACCEPT

What's this up to? Allow anything in to the FTP control port if it is part of a NEW or ESTABLISHED connection.

Then allow anything in for anything at all which is RELATED.

But what is RELATED? This is where the nf_conntrack_ftp module comes in. Just by loading this module, IPTables sits there looking at every incoming and outgoing packet and inspecting at a high level for FTP commands and tagging the packets. If a packet comes in for a PORT command which has been agreed as part of the original port 21 FTP Control connection, it is tagged as RELATED and the RELATED match / rule lets it back in. It really is that easy.

While we're here, let's cover setting up a nice, secure FTP server for web / vhost access:


 * apt-get install vsftp
 * Edit /etc/vsftp.conf: turn off anonymous.  Set umask to 022.  Allow "normal" user login.  Set chroot option.
 * Edit /etc/pam.d/vsftp, comment the first "pam_listfile" line and add: 
auth    required    pam_listfile.so item=group sense=allow file=/etc/ftpgroups onerr=succeed
 * Create /etc/ftpgroups, add the single line: ftpusers
 * Add system group ftpusers: groupadd ftpusers
 * Add a user and put them in the group: useradd myftpuser; adduser myftpuser ftpusers;
 * Edit /etc/password and set "myftpuser"'s home dir to the appropriate vhost
 * Set perms on vhost directory such that it is writable by your user
 * Set a password for myftpuser
 * Restart vsftp
 * Edit /etc/fail2ban/jail.conf: in Ubuntu there is a section for vsftp already, just set "Enabled" to "true.  This converts multiple failed attempts to authenticate to firewall blocks, stopping kiddie scripts
 * Restart fail2ban
 * Test - ftp in both Active and Passive mode and make sure you can GET and PUT files.  Check also that no other users can log in.

So now you have vsftp configured such that only members of the ftpusers group can log in, and when they do, they will be stuck in their own home dir - the vhost they can access - and any files they upload will automatically be world readable. Also, fail2ban will stop anyone trying to brute-force your server.

Enjoy.

...Click for More

Article

iptables

nf_conntrack_ftp

secure FTP

vsftp

Compressed, VPN'd Remote Desktop over SSH

If you often Remote Desktop in to Windows servers over a Broadband connection, this might help you out a bit. Or if you look after various clients and have only key-based SSH access to their network (as you should), this will also be handy.

Sometimes you might find your RDP session a bit sluggish, and having RDP open to the Internet isn't really such a great idea. You might also need to administer many Windows servers, but naturally you don't want to port forward for different servers.

If you've got a Unix box on the inside, here's what you do.

Let's assume your Unix box has SSH set up for key based auth, because it is most secure and easily automatable. And, of course, SSH is forwarded in through the firewall.

Edit your local .ssh/config file:

ServerAliveInterval 30
CompressionLevel 5

ServerAliveInterval means that the connection will have some data passed across it in each direction every 30 seconds. This is especially useful for me because my cheap ISP supplied router likes to randomly drop connections it thinks are idle. Every. Five. Minutes.

CompressionLevel is the "gzip" compression applied, 0-9. 5 is a good compromise between CPU cycles and a responsive session. Compression is the key to this.

Now, set up your SSH connection to forward:

ssh -C -N -L$SOURCEPORT:$TARGETIP:3389 $UNIXBOX &

What does this do, exactly? It SSH's in to $UNIXBOX and sets up a forward, so any locally generated traffic to $LOCALPORT is pushed down the SSH connection and forwarded to $TARGETIP on port 3389.

-C turns on Compression. -N tells it to not bother starting a remote shell on $UNIXBOX - it is a forwarding connection only.

Now to see how well a remote desktop session responds running over this SSH VPN:

rdesktop -g 1680x950 -z localhost:$SOURCEPORT

-g sets up the size (geometry) of the window.

-z tells it to compress the connection. To be honest, I've never noticed a difference using this or not. In theory, it is compressing everything twice which shouldn't help. But overall, using this article, your rdesktops will be more responsive, and this option seems harmless.

Now, what if you want to connect to more than one RDP server at a time? Simply use a different $SOURCEPORT in the above two lines.

You can easily script these things together, and even use netstat cleverly to pick a random and unused $SOURCEPORT.

Enjoy.

...Click for More

Article

Port forward

RDesktop

SSH

VPN

Simulating "bind" for IE8 using "call" and "apply"

Isn't it fun having to learn two (or more) different ways of coding everything, just to be able to support Internet Explorer?

I discovered this morning that IE8 doesn't implement that most wondrous of typing and sanity savers, "bind".

Quick recap: "bind" is a method on a "Function" object which returns a function bound to a context of your choosing.

Most commonly, I use it when attaching object functions to events, such that they will execute in the context of this object. This is probably best illustrated with a highly contrived example:

var x = {
  foo: "Hello World!",
  fum: function()
  {
      console.log(this.foo);
  },
  bar: function()
  {
      $("b.clickity1").click(this.fum.bind(this));
      $("b.clickity2").click(this.fum);
  }
}
x.bar();
window.foo = "Goodbye cruel world!"
$("b.clickity1").trigger("click");
: Hello World!
$("b.clickity2").trigger("click");
: Goodbye cruel world!

What's it do? We have an object with a property "foo" and a method "fum" which puts foo's value to the console log.

Another method, "bar", sets up a click handler (using jQuery) for some item on the page. But we don't assign this.fum directly to the handler, we assign the function returned as a result of binding it to this object. That means that when the click event calls our handler, the handler's "this" object is...this object.

A second click handler for another DOM object is set up with just this.fum passed through as the handler.

When we trigger the two click handlers manually, you can see the difference: In the first case, the handler has been called in the context of the x object. In the second, it has been called in the context of the window object. Or to put that a different way, the first click handler will force bar to be called with a "this" of "x", the second with a "this" of "window".

So you can see, bind makes it very easy to stay inside our private object context, however the handler is called. You might use this in an object which sets up, say, a button toolbar, where the toolbar calls in to a blog object to save, cancel, embolden words etc.

But the genius simplicity of this bind call is missing from IE8 and before.

Luckily, "call" and "apply" are still available. Similar principals are at work with "call" - every function has a "call" method, who's first argument is the context to use. So in the above example, instead of:

$("b.clickity").click(this.fum.bind(this));

You would do this:

      var me=this; 
      $("b.clickity").click(function(){me.fum.call(me);});

Which is clearly a little more laboured and less readable. It has its own uses though, and in some cases is more appropriate than bind.

"Call" takes a variable number of arguments and passes them straight through to the function unchanged:

me.foo.call(me, 3, 5, 7)
...
foo : function(prime1, prime2, prime3)
{
   ...
}

"Apply" is very, very similar to Call, except that it takes only two arguments: the context and an array of arguments to pass to the function:

args = [3, 5, 7] me.foo.apply(me, args) ... foo : function(prime1, prime2, prime3) { ... }

There is some more fun you can have with bind: you can pass in a variable number of arguments, which will be given to your bound function whenever it is called, before any others.

So where a click handler will normally be passed an Event object:

foo : function(e) { ...do stuff... }

You can arrange for things which are only in context at bind time to be given to the handler:

$("b.clickity").click(this.fum.bind(this, 7));

foo : function(mostMagicalNumber, e) { this.magic = mostMagicalNumber; }

Notice the mostMagicalNumber comes before the Event object (e) that was passed at "call time".

Let's draw these things together to give us a usable "bind" in IE8 (And, in fact, on some mobiles):


if( ! Function.prototype.bind )
    Function.prototype.bind = function(context)
    {   
        var args = Array.prototype.slice.call(arguments, 1); 
        var me = this; // the function being called
        return function()
        {   
            return me.apply(context, new Array(args, arguments));
        }   
    }

Enjoy.

...Click for More

Article

bind, call and apply

Internet Explorer

javascript

Programming

Internet Radio

Keep those long, dark, winter nights bouncy.

Code South:
http://codesouth.fm/
( Local / Brighton dance station - various local DJs, mixture of music)

DnB streams:
http://relay-ntc0s-5.shoutcast.net:8030/
http://dnbheaven.com:8000/
http://stream.247drumandbass.com:2470/
( All from "Digitally Imported" - MASSIVE collection of radio streams:
http://www.di.fm/
Look on the HUGE "Listen Now!" menu )

Also:

http://www.flightfm.co.uk/

http://www.infront.fm/

http://www.subbassfm.com/

PartyVibe Breakbeat stream:
http://www.partyvibe.com:8012/
( From http://www.partyvibe.com/ - various stations: Breakbeat, Trance, Dubstep, hardcore )

Groove Salad:
http://somafm.com/play/groovesalad
( Part of SomaFM, lots of channels: http://somafm.com/ )

...Click for More

Ambient

Article

Breakbeat

Code South

Digitally Imported

Drum and Bass

Music

Radio

http://www.youtube.com/watch?v=lYSX51xBkos

...Click for More

Interesting

Lucid Dreaming

Video

YouTube

http://newsthump.com/2012/10/04/bbc-executives-admit-they-knew-dj-moyles-was-shit/

...Click for More

Article

Chris Moyles

Funny

News

Newsthump

Spoof

I really need to build these book shelves to annoy someone...

http://imgur.com/a/9fJLP

...Click for More

fun

Gallery

Image

imgur

OCD

Shelves

Forwarding a VMWare connection over SSH

Say you have a remote site to support with a VMWare setup and no direct VPN, how can you check or fix anything on their main VM hosts?

If they have a local Linux or Unix box, there is a way. You can forward all the required VMWare ports over SSH...

sudo ssh -L*:80:192.168.5.8:80 -L*:443:192.168.5.8:443 -L*:902:192.168.5.8:902 \
         -L*:5989:192.168.5.8:5989 -L*:8000:192.168.5.8:8000 \
         -L*:8100:192.168.5.8:8100 adminuser@ssh.myclient.com

This relies on the client's firewall having SSH fowarding to a local Linux/Unix box, say 192.168.5.67. And this example has ssh.myclient.com pointing to their external IP, of course.

So, what are we doing here? Run this on a local Linux box (say, 192.168.23.80), and it will SSH to adminuser@ssh.myclient.com. Local port forwarding is set up for ports: 80, 443, 902, 5989, 8000, 8100.

The fowarding is such that any connections to these particular ports on 192.168.23.80 from any PC on your local network are sent over the SSH connection to the remote SSH server you have connected to. From there, the connections are pushed out to 192.168.5.8. The VMWare Host doesn't know the difference and just accepts the connections as if they were local.

The * at the front of each "forwarding option" is what works this magic - it instructs SSH to listen for traffic to forward on ANY external interface.

So now, all you need to do is point your VMWare/VSphere Client at 192.168.23.80 and pop in the username and password, and you have full control of your client's VM Host.

VMClient ---> Local Linux box (SSH) ---> ((INTERNET)) ---> (SSH) Remote Linux box ---> VMHost

Enjoy.

Important: If you set yourself up to be able to do this, allowing anything on the net to SSH to your server, be aware of how much control it would give an attacker over your network if they could guess your password.

Lock out unneeded, guessable accounts from login, such as 'root'. Use fail2ban to stop brute force attacks. Use long passwords. Or, if possible, don't even use passwords at all - use ssh key based authentication.

...Click for More

Article

Port forward

Remote

SSH

VMWare

Outlook 2010 very slow starting

I recently upgraded a client to Office 2010.

Amongst the many teething troubles were a couple of users complaining that going in to their eMail was now very slow. The problem looked like this:

Start Outlook
Minute or two pause, then a certificate warning about "autodiscover.myclientdomain.com" not matching the cert
Up to 20 minutes of hourglass, frozen Outlook. Then it started working.

Well, the fix was actually very simple: Outlook 2010, unlike previous versions, scans for new mail on startup in every account and pst.

If you have a missing Archive PST (Maybe you archive to a PST on an external drive which isn't currently connected) or you have been granted access to another user's eMail, and they have since left and been removed from Exchange, Outlook 2010 will hang for a stupidly long time before timing out. But without doing anything sensible, such as showing you an error message. You've got to love Outlook, eh?

The fix turned out to be fairly simple: Remove the other user's mailbox from this Outlook. (Or remove the link to the Archive or make the archive.pst available, if this is the problem)

Enjoy.

...Click for More

Article

Office2010

Outlook

Slow start

Configuring your Javascript App from PHP

How often have you been writing a Javascript front-end to a web application and thought "If only I knew how the CMS I'm using on the server is configured?" or "What page does my CMS think I'm on?" or "Does the CMS know I'm logged in? Should I create and show some Admin controls?"

There may be debug options, current page name, hints about what the user has done recently, any number of things which it would be useful for your Javascript app to know.

You don't want to change code in production; that must always be a snapshot of the current tested system. You also don't want to have PHP adjust the Javascript as it is served as this becomes tricky, unreadable and, above all else, slow.

So how do we do this? Well, this is one way.

You can store up to 65Kb of data in an HTML attribute. You can put more anything you like in there. So what we're going to do is BASE64 encode a JSON encoded associative array and hide it as an attribute in a hidden "bold" tag just before the closing "body" tag, something like this:


$jsOptions = array("BASEURL"=>BASEURL,
                                  "DEBUGJS"=>DEBUGJS,
                                  "Page.Name"=>$env->page->Name
                                  "IsLoggedIn=>$auth->isLoggedIn());
$jc = json_encode($this->jsOptions);
$jc = base64_encode($jc);
$before = "<b id='WIT_JSBOOT' style='display:none'" .
                  " conf='{$jc}'></b>";
$html = str_ireplace("</body>", $before . "\n</body>\n", $html);

By JSON encoding it, we can so very easily extract it the other end. Base64 encoding guarantees no problems with odd chars or charsets.

So now you have some HTML that may look a little like this:

...
<b id='WIT_JSBOOT' style='display:none' conf='eyJCQVNFVVJMIjoiaHR0cHM6XC9cL2Rldi5vbGljb21iZXIuY28udWtcL29saWNvbWJlci5jby51a1wvd2ViXC8iLCJERUJVR0pTIjp0cnVlLCJQYWdlLk5hbWUiOiJob21lIiwiaXNMb2dnZWRJbiI6IiJ9'></b>
</body>

So how do we access this from Javascript? We just need to get hold of the "conf" attribute from this node and decode it:

$(document).ready(function()
{
   // Naturally, we wait till the DOM is ready so we can get at our config.
   // Things that need to know about config need to check it exists and try again later.  Or catch the
   // "Config.Loaded" event
    var conf = document.getElementById("WIT_JSBOOT")
                      .attributes
                      .getNamedItem("conf")
                      .value;
   conf = atob(conf);
   window.config = jQuery.parseJSON(conf);
   // Fire off a custom jquery event to anything that might be interested in this being ready
   $(document).trigger("Config.loaded");
});

(You'll want to search the 'net for "atob" and "btoa" - at the time of writing, these only work in Firefox, Chrome, Opera - but there are many implementations available to use in Internet Explorer)

And hey presto, any of your Javascript can now access a global object on the Window called "config", which in this case looks a little like this:

conf := {BASEURL: http://olicomber.co.uk/, DEBUGJS: true, 'Page.Name': 'home', isLoggedIn: true};

From here, you can now easily create page or webservice URLs pointing back to the proper place, or show controls which should only be seen if the user is logged in. You could even publish the "User Type" and other info here and make the interface completely personalised. (Though obviously these are "hints" and not "security" - any calls to the server should be validated and authenticated there)

Enjoy.

...Click for More

Article

configuration

javascript

json

php

Programming

Multiplexing ALSA input streams with DMIX

You may have noticed on a stand-alone Linux server (no GUI) that only one device at a time can use the sound card. This is deeply annoying and frustrating. It turns out you don't need to mess around with Pulse, Jack or OSS4 to get different sound input streams working, you can use ALSA, but its default config needs tweaking.

The problem is that your ALSA loving sound programs may latch on to the "default" device, which doesn't multiplex. The first program to open the default device locks it. What we need to use is the dmix ALSA plugin.

So we set up the DSP device all these apps are using to secretly pass everything through to DMIX:

File /etc/asound.conf:

pcm.dsp {    
    type plug    
    slave.pcm "dmix"
}

And that's it! Restart all your sound apps (in my case MPD and VLC/sdl) and they should now play together smoothly.

...Click for More

ALSA

Article

dmix

multiplex audio

https://www.youtube.com/watch?v=DkaUsBwe0fo

...Click for More

Geeky

Gotye

Music

Video

YouTube

http://soundcloud.com/doctor-moody/rock-it-surgery-drmoody-live

...Click for More

Audio

Breakbeat

Funky

Mix

Music

SoundCloud

http://www.youtube.com/watch?v=DrLH1h78KhI

...Click for More

Cats

Funny

Video

YouTube

Easy, reliable spam fighting with Exim

Exim, as fans of Postfix, Qmail or, god help us, Sendmail, won't tell you, is a fantastic bit of kit. The way it gives you absolute control over what happens to an email at every step of the way, the way it runs through a dynamic, living, breathing config file for each individual email just makes sense and is a joy to set up.

Well, some of this isn't strictly true: the learning curve is steep. Today, I'm going to take you through my standard Exim config file and show how you have all the tools available for some seriously good, low maintenance, low cpu spam fighting.

Download the whole config file: FileNotFound: /home/vhosts/olicomber.co.uk/web/downloads/exim4.conf

Now, why would you want to do any specific spam fighting in your mailserver? After all, you have a perfectly good spamchecker to pipe it out to. Spamassassin, for instance, is brilliant software. The answer is that if you do a few simple checks on the email first, you may not need to spend all those CPU cycles on properly checking it, and if you run a mailserver of any non-trivial size, this will save you many headaches.

Also, where an email is borderline spam, you may be able to tip it in the right direction, avoiding both false positives and false negatives. I don't know about you, but I hate calls from my customers asking why a particular message has been classified as spam when it "clearly isn't"...

You may have noticed similarities in spam you receive: the computer it came from wasn't a mailserver, it certainly wasn't the right mailserver for the sender, it doesn't have reverse DNS, it's on a blacklist or two, it has no SPF record. These are simple checks if you have the information at your fingertips. Why go to all the expense of invoking another program to do all this for you?

Of course, I am not advocating not using a spamchecker. But if all the simple tests you can do on information you have to hand fail, why carry on processing? Also, if some of these tests fail, and they are different tests to your spam scanner, how can we make them count?

Let's step through the config and see how this works. There will be a lot of other information on Exim along the way, to give background and context.

There are 5 sections in a typical config file:

Globals
ACLs
Routers
Transports
Retry
Rewrite
Authenticators

We will look at these in order.

Globals

The first section of the file we will largely ignore; this is an overview of Exim with an in-depth look at spam fighting. There are many great books on Exim and it would be pointless to attempt to reproduce this rich wealth of information here.

But briefly, it starts with the SQL queries which make my Exim work as a virtual mailserver. The users on my mailserver don't, in an operating system sense, exist. But they also have a lot of attributes which the operating system wouldn't be able to tell us about if they were real users. But notice how, in the queries, there are non-SQL strings such as '${quote_mysql:$local_part}'. There are three clues here as to Exim's power. One is the substitution itself and the second is the "$local_part" bit which shows that Exim pulls out and makes available as variables lots of useful things about the email itself, in this case the first part of the delivery address. The third is a little more subtle - what is $local_part when you write the config? Answer: Nothing. The config file is evaluated and expanded every time a new email is dealt with.

It is this constant re-evaluation and expansion that is the key to Exim's power.

Also worth noting is how easy it was to create these substituion macros - we now have, for instance, a macro MYSQL_Q_BOXNAME which expands from certain information we know about the email - the domain and user it is addressed to - to the location of that user's Mailbox (if the user exists!)

We'll see later how easy it is to use them, and how easy it is to create general variables to query and compare.

ACLs

The ACLs are where the magic happens. I'd love to meet Philip Hazel and shake his hand, as these ACLs are one of those "I could have thought of that" ideas that are the mark of genius. No one else had thought of it, but in hindsight it seems obvious: Exim treats each each email as if it were a network packet passing through a firewall.

Let's recap an SMTP conversation. SMTP goes back many years and was implemented as a text protocol, I presume so that one could communicate with a mailserver using a telnet session. So if we telnet to a mailserver's port 25, this is what the conversation will look like:

Server: 220 mail.olicomber.co.uk ESMTP Exim 4.76 Sat, 22 Sep 2012 15:14:31 +0000 Me: HELO mail.somedomain.com Mailserver: 250 mail.olicomber.co.uk Hello mail.somedomain.com [86.30.168.41] Me: MAIL FROM: someone@somedomain.com Mailserver: 250 OK Me: RCPT TO: oli@olicomber.co.uk Mailserver: 250 Accepted Me: DATA Mailserver: 354 Enter message, ending with "." on a line by itself Me: Hi Oli, how you doing? . Mailserver: 250 OK id=1TFRSP-0007Nr-LC Me: QUIT Mailserver: 221 mail.olicomber.co.uk closing connection Connection closed by foreign host.

Notice the commands being used: (connect), HELO, MAIL FROM, RCPT TO, DATA and QUIT. Each of these is a stage the message has to go through, and different amounts of information are available at each stage. (connect) is a kind of virtual stage - the server knows we have connected but we haven't asked for anything to happen yet.

At CONNECT stage, only our IP is known to the server. HELO doesn't give much more information to the server, though it is expected to be "well formed". MAIL FROM gives an email address - a "local part" and "domain name". And so on.

The ACLs correspond to these stages of the email. Each ACL acts as a firewall: A set of rules must be passed, one at a time. Each rule can ACCEPT or DENY the message. If it is ACCEPTED, we are done with this ACL and move on to the next. If is REJECTED, processing of this message will stop and an error will be shown to the person connected.

If all ACLs pass, the message is queued up for routing. This is covered later, but for now picture it as Exim is happily sitting on the message and may delivery it, forward it, send it back...or something even more exotic.

Time to look at the spam fighting specifics in the ACLs...

The basic jist is that we start a counter at 0, increment or decrement it according to various conditions as the mail goes through all ACLs, and finally deliver it to inbox or spamfolder depending on the value of the counter. If the counter is excessive at the end of the ACLs (some spammers are shockingly bad at their jobs, one wonders if their hearts are truly in it) then the message will be dropped and the spammer told to go away.

So, to start with, we initialise the counter in the CHECK_SMTP ACLs. There is nothing else we can usefully do in this ACL, so it is an ACCEPT rule:

accept set acl_m69 = 0

What is interesting about the "acl_mXX" variables we set is that we can manipulate them in similar ways to variables in other languages, unlike the SQL expansion macros we saw above, and they will exist and hold their values until Exim has finished dealing with the entire message - including delivery.

Let's walk through the check_mail ACL. At this stage, we have an IP of the server wanting to send and an email address for the sender. We can do a surprising number of checks on this info.

Out there on the internet, where servers are large and sites are small, it is often the case that a single server will deal with all incoming and outgoing messages. The mailservers you will think of straight away are exceptions to this rule - gmail, hotmail, yahoo all use arrays of incoming and outgoing servers. But many, many other sites just use a single server.

A mailserver for a domain is identified by an MX Record. So, if we can establish that the computer talking to us is the mailserver for the domain the email is coming from, we can put a little more trust in it and not be so quick to spam it. Remember, it is illegal to send spam, therefore spammers don't send from real addresses or mailservers.

So we test for this, using a rather complex couple of rules:

 warn set acl_m0 = ${if match { ${lookup dnsdb{A=\ 
                                 ${sg {${lookup dnsdb{mx=$sender_address_domain}{$value}{}}}{[ \n]}{:} } }\
 {$value}{}} }\ {$sender_host_address} \ {1}{0}} 
warn !condition = ${if eq {$acl_m0}{1}{1}{0}}
 # Increase the spam score if it's not the proper MX sending it 
set acl_m69 = ${eval:$acl_m69+20} add_header = MXCheck: Server is NOT MX for sender's domain, +20 Spam score

What the above does is two nested lookups to establish the IP addresses of the MX servers for the senders domain and compares with the IP address of the connected computer. The second rule checks the result of the first, adjusts the counter and adds a header to the message. If the test fails, 20 is added to the spam counter.

In this way, an email which has a "from" address of bob@examplesite.com but which is coming from the IP of someothersite.com's MX record will be hindered and is more likely to later be classified as spam.

Now we have a look at SPF. SPF is a special kind of DNS record: If we look up the senders domain and it has an SPF record, it will tell us all the addresses of the servers which are allowed to send email for this domain. This is related to the first test, but is more accepted: one may have several MX servers and several sending servers, which are all different. In this case, our first test above fails, but we are still talking to a valid server, so the mail must be kosha (or we can, in theory, prosecute the sender)

warn set acl_m1 = ${run{/usr/bin/spfquery --ip $sender_host_address --id $sender_address}{1}{0}} warn condition = ${if eq {$acl_m1}{1}{1}{0}} set acl_m69 = ${eval:$acl_m69-30} add_header = SPFCheck: Server passes SPF test, -30 Spam score

If SPF passes, we subtract 30, thus undoing any negative waiting above due to it not being an MX server, and weighting the email in favour of not being spam. If you use this configuration, you may feel this weighting should be higher, and there is no reason it couldn't be. 30 is quite conservative.

What I like to do at this point is have a little fun with the spammers. A spammer tries to send as fast as possible, often using a non-standard program to rattle through thousands or millions of email addresses. If we decide at this point the email might not be kosha, we throw in a couple of seconds delay. A lot of the worst spammers will disconnect if they get too delayed.

  warn set acl_m2 = ${if or {{eq {$acl_m0}{1}} \
                            {eq {$acl_m1}{1}}}{1}{0}}
  accept 
    condition = 1
    delay = ${if eq {$acl_m2}{1}{0s}{2s}}

Any well configured server will have reverse DNS. A home PC will not. There are, unfortunately, a lot of badly configured servers out there. If no reverse DNS, add a bit more spam score:

  warn 
    condition = ${if eq {$sender_host_name}{} {1}{0}}
    set acl_m69 = ${eval:$acl_m69+20}
    add_header = ReverseDNS: No reverse DNS for mailserver, +20 Spam score

We don't penalise too badly at this point due to the number of badly configured servers. We are simply helping the weight up. If the message eventually turns out to not be spammy at all, this won't matter. But if it is spammy, it will tip it over the edge.

And that is the end of the check_mail ACL. The ACCEPT above with the delay means that all mail reaching that point will move to the next stage of the SMTP Protocol and its matching ACL: RCPT TO.

The first thing to do is check some blacklists. Spamhaus is a good one.

  warn 
    dnslists = zen.spamhaus.org
    set acl_m69 = ${eval:$acl_m69+30}
    add_header = BlacklistCheck: Blacklisted address, +30 Spam score

In actual fact, this check could possibly go in the check_mail ACL as it is based purely on the sender; I cannot remember why it is here. A failure here does not block the mail. I've been the victim of blacklists before and I think it is quite unfair to block based on this, when it is so easy to get on a blacklist. So instead, we add some more spam score.

  warn
    set acl_m70 = ${lookup mysql {MYSQL_Q_AUTOWHITELIST}{$value}}
    set acl_m69 = ${eval:$acl_m69 + $acl_m70 + 20}
    add_header = Autowhitelister: awarded +${eval:$acl_m70+20} Spam score

Here's a fun one - autowhitelisting. If you remember in the first section, there is a query denoted by the MYSQL_Q_AUTOWHITELIST macro which calls a function with the sender, recipient and IP address. Every time this function is called, it decrements a row in a table which is key'd by sender, recipient and IP address. This query returns the value and adds it to the spam score.

The net result is that the more times one person emails another, the more likely it is to get through spam scanning.

...and that is the end of the check_rcpt ACL. Next, we move to the DATA part of the protocol, and run through the check_data ACL.

The DATA part is the body of the message. This is looked at by very, very clever spamscanners - in my case, Spamassassin. But only if it is worthy.

Spammers do not generally send large messages, so we do not spam scan if the message is larger than 200k. This saves much expensive processing of a large message.

Also, we look at our counter. If our counter is already over 55, we know we will be classifying it as spam anyway, so simply do not bother scanning it. This is where the cpu cycles are greatly saved.

  warn 
    condition = ${if and { {< {$message_size}{200K}} { <{$acl_m69}{55}}}{1}{0}}
    spam = nobody:true/defer_ok
    set acl_m69 = ${eval:$acl_m69 + $spam_score_int}
    add_header = SpamScan: awarded +$spam_score_int Spam Score
    add_header = SpamScan: Report: $spam_report

  warn
    add_header = SpamTally: Final spam score: $acl_m69

The final WARN adds a header to the message so the recipient can easily see (if they "view source" on the message) why it was classed as spam.

  drop
    condition = ${if >{$acl_m69}{300}{1}{0}}
    message = Your message was classified as SPAM.  Please add more content, cut down on HTML links, use fewer naughty words etc.  Also, ask your IT dept to make sure your mailserver has REVERSEDNS, SPF and is not on any black lists. Your score: $acl_m69

Finally, if the message was stupidly spammy, simply drop it. There's not point even filing it in the spam folder.

At this point, we know the mail is ok to deliver so we ACCEPT.

Routers

Routing in Exim is what happens if a mail successfully negotiates all the ACLs. It determines where the mail is sent.

A Router has various conditions attached to it. If it matches the mail being processed, its action is taken. This usually means invoking a transport to do the meat of the work. Just like the ACLs, the Routers are run in order until one is matched. If there is no match, the mail is rejected and a reject message will be sent back (unless configured not to)

A good example is the remote_smtp router:

dnslookup:
  driver = dnslookup
  domains = ! +local_domains
  transport = remote_smtp
  ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8
  no_more

This says that it is a dnslookup router. If the domain is not in our list of local domains, if it is not targetted at localhost, then give it to the remote_smtp transport to deliver. remote_smtp will do as it sounds and dial up the remote mailserver in question to deliver it.

We are going to look at the local delivery routers.

thisisspam: driver = accept domains = ${lookup mysql {MYSQL_Q_LOCAL}{$value}} local_parts = ! root condition = ${if >{$acl_m69}{55}{1}{0}} headers_add = Filed as SPAM - Final score ($acl_m69) greater than 55 transport = spamfiler no_more local_deliver: driver = accept domains = ${lookup mysql {MYSQL_Q_LOCAL}{$value}} local_parts = ! root transport = local_deliver no_more

The local_deliver router simply accepts the message if the user exists. The MYSQL_Q_LOCAL macro will expand to a domain if the user exists. The transport used will be local_deliver.

The thisisspam router is very similar, but has a condition attached. The router matches iff the spamscore is greater than 55. In this case, it hands the message to the spamfiler transport and adds a header to the message.

Transports

The transports are configurable modules which do some work with an email to deliver it locally or remotely or perhaps produce a vacation message. There's all sorts that can be done here, and I recommend a good read around the Exim website.

Unlike ACLs and Routers, Transports do not run in order; they are called by Routers.

Here we will look at the two local delivery transports.


spamfiler:
  driver = appendfile
  directory = /home/email/${lookup mysql {MYSQL_Q_BOXNAME}{$value}{root}}/.Spam
  user = vmail
  group = vmail
  maildir_format
  mode = 0660
  directory_mode = 0770
  create_directory


local_deliver:
  driver = appendfile
  directory = /home/email/${lookup mysql {MYSQL_Q_BOXNAME}{$value}{root}}
  user = vmail
  group = vmail
  maildir_format
  mode = 660
  directory_mode = 0770
  create_directory

I won't dwell too much on what all this means; I just wanted to highlight the difference between them. The local_deliver transport uses appendfile to place the email in the top of the Mailbox, which is the inbox, whereas the spamfiler places the email in the .Spam subfolder.

Well, that's it. I hope that has given you some good ideas about the insanely useful role your mailserver can take in weeding the ham from the spam.

If it has converted you to Exim, so much the better :-)

Enjoy.

...Click for More

Article

Exim

Spam fighting

Whitelist

Using Dropbox for Windows Laptop backup

If you have spent much time in IT support, you'll know keeping your remote workers' laptops backed up is a nightmare.

You can give them an external HD, they won't use it. You can set up backup software on it, they'll never remember to run it. You can schedule it to run, it won't be switched on at the right time. You can schedule it for during the day and they'll complain everything is slow. The worst part is, you may not even know anything is wrong until you need that backup. You can't win.

Or can you?

Here's a workable, reliable, simple backup solution for laptops, using DropBox*.

Install Dropbox. Configure it with the Dropbox folder somewhere with lots of room, perhaps D:\ if you have one, or C:\. Do not put it in your Docs dir.
Set up Folder Redirection: http://www.virtualizationadmin.com/articles-tutorials/terminal-services/performance/configure-folder-redirection.html

The trick when redirecting is to use "\\localhost\c$\dropbox"

Hey presto, all files you "Save to Desktop" or "My Documents" are transparently saved to C:\dropbox, where they are "instantly"** sync'd to your Dropbox space.

* Or Google Drive... or any other similar "Cloud Storage" service. Here's a nice list of similar Dropbox type projects: http://www.webresourcesdepot.com/open-source-dropbox-alternatives-to-start-building-a-file-storage-sharing-system/

** Provided there is a network connection. If they can't be sync'd now, they will be sync'd later.

...Click for More

Article

Backup

Dropbox

Laptop

https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-snc7/405780_469432469744436_113598622_n.jpg

...Click for More

Funny

Image

Jpeg

Mars

Contact

Send me some abuse, it keeps me in check.

...Click for More

Article

Fixme

Recent articles