Paul Gregg

Jack of all Tech.

Is BT still collecting usage data?

Written By: pgregg - Mar• 31•2010

After the recent scandal over BT using Phorm (see here and here) is it still possible that BT is still covertly collecting data profiling its user’s internet usage?

I have, as part of my home BT Total Broadband package, a HUAWEI Mobile Connect 3G Modem.  This is a USB dongle that provides internet service.   I rarely use it, but because of the storms last night my internet connection has been very unstable, therefore I plugged it in today.

When I started the “BT Connection Manager” software, it informed me that there was an update to the service and would I like to download it? Yes, ok, and a couple of minutes later it began to install itself.  Then popped up the obligatory EULA which contained the paragraphs:

Installation of this Software will automatically apply a unique identifier to the Software, this identifying feature will only be used in accordance with the BT privacy policy which can be found at www.bt.com.

This software automatically collects session and client parameter Information for all connections made via this Software. This information is automatically transmitted to BT at the start of each session where Internet access is available. This session information will only be used in accordance with the BT privacy policy which can be found at www.bt.com.”

BTConnectionManagerEULA.txt

I also note that the software package uses the Open Source packages libCurl, openssl and sqlite.  Combined they enable the package to record “session and client parameter Information for all connections” in the database (sqlite) and to encrypt (openssl) and transmit the data via a web request (libcurl) back to BT.

At this point I am unable to confirm if such a transmission is taking place as I declined the EULA, but would welcome feedback or comments from others that are able to investigate this more fully.

Update: Even though I clicked “Cancel” to reject the EULA – it did update the application and actually started it up before I exited the installer.  Guess I have this privacy invading feature now.

Amazon 1 – 0 Waterstones, aka this morning’s rant.

Written By: pgregg - Sep• 29•2009

SWMBO asked me to order a book for her this morning, so being the tightwad that I am, I go hunting for the ISBN and the cheapest place to buy it.
Amazon has it for £17.24 and Waterstone’s for £18.49 (but if I order through quidco, then I receive another 8% cashback, making Waterstone’s cheaper).

Waterstone’s need me to create an account, fair enough and state “Also, please be aware that passwords are case sensitive, and must be a
minimum of seven characters long and at least one character must be
numeric.”  Again, all fairly standard.

So each time I try to enter a password which meets this criteria I’m hit with a javascript popup that claims: “Your new password must be at least 7 characters long and contain at least one digit.”

“But it is!”, I yell in frustration.

Using the Firefox “Web Developer” plugin I unhide the password boxes and see that my password doesn’t have digits… wtf!   I enter the digits again and find there is another hidden rule:

<input name="newPassword" type="password"
id="fPassword" size="40" maxlength="16" value="" />

Yes, an upper limit of 16 characters on the password.

Right ok, make a password of 16 chars or under and move on…

Add name, postcode to the next form and it finds my address ok – click continue to (I presume) proceed to the CC entry form and voila!

Well not quite.

“There has been a problem processing this request

Please use the refresh button on your browser to try again.

Thank you.”

A few refreshes later, it is apparent that Waterstone’s has no intention of working, sod it, for 20p more I can have less hassle at Amazon.

Getty Images / BBC Math Fail.

Written By: pgregg - Jun• 03•2009

From the BBC Magazine today is an article 7 questions on GCSE maths.

Can you spot the fail?

bbc_math_fail1

Yes, epic.

bbc_math_fail2

Register your broadband not-spot now

Written By: pgregg - May• 28•2009

thinkbroadband, formerly ADSLGuide.org.uk, have setup a community site where UK sufferers of the lack of universal broadband can register their inability to obtain broadband (a not-spot) or where you are unable to obtain up to 2Mbps service (slow-spot).

The Northern Ireland map is here – if you currently suffer from a lack of, or poor broadband coverage (including satellite) please register your details on the site – by adding your postcode it will appear on the map and we can begin to get a true picture of the poor state of broadband in Northern Ireland.

Thanks.

Borland, Farewell my sweet.

Written By: pgregg - May• 06•2009

Today, May 6 2009, marks the day that Borland, that once great master of all software development has finally recognised there was no other recourse but to up and sell itself off in order to survive.

Back when I was a teenager, in the early 80s and personal computing was coming to the fore – I, and many others, aspired to work for that great company Borland. It was the pinnacle of language development and development tools and we wanted to work there. However, based in Ireland it was never to be.

Also, once upon a time I happened to be working for a very promising young company with a fantastic product line called Segue Software, based in Boston, MA.  Segue also had its troubles but a new CEO saw its fortunes turn and it was climbing to success. This success was noticed by the aforementioned Borland as it tried to re-invent itself as an Application Lifecycle Management (ALM) company.  The same day it was announced that Borland was acquiring Segue, it also announced it was selling its developer tools division (that’s Delphi, JBuilder, and later Delphi for PHP, 3rd Rail line of products).

This was such a bitter-sweet time for many.  I was overjoyed that I was going to work for Borland (childhood dream) – it didn’t matter I wasn’t going to work with the developer tools, working for “Borland” would just be cool.  Sadness also because our little 200-man company was being consumed by a 1200-man behemoth (relatively) and no-matter which way you looked at it, people were going to lose their jobs.  Pretty much the entire US East Coast staff (Segue Head Office) lost their jobs and the office was closed.  Product development labs and Technical Support survived, simply by virtue that it was the product and product skills that were purchased, not the G&A functions – they could go.

I ‘lived the dream’ for the next 2.5 years in the IT department. Despite being remote, I loved working with the rest of the Borland teams as I was intimately involved in the merging/migration of Segue’s systems into Borland’s. I also had the pleasure of working with several departments to architect and deploy several new platforms (such as product downloads and licensing via Intraware, and the companies Salesforce.com, SFDC, deployment).  I’ll treasure the time I spent at Borland.

Of course there were several WTF moments. Most significantly, for me, was the company “hanging its hat” on BMS (Business Management Solutions) which ultimately proved to be a hatstand made of jello. Very few, outside of management and that product team, believed in it.  Another significant WTF for Borland was, If you plan to be the Application LIFECYCLE Management company – why divest yourself (for a paltry $27m) of two of the world’s major AppDev toolsets (Delphi and JBuilder). You’ve just removed the feeder market and upsell opportunity into your ALM business. Finally, and internal WTF to get off my chest, on what planet does the IT department belong as a subdivision of the HR department?

Borland will live on in the hearts of many of us who knew what she used to be. I think I left Borland a better place than I found it (as long as you don’t look at the stock price ;), and I made some good friends. At the end of the day, there isn’t much more you can ask from your tenure.
It is sad that today if you ask a typical Software Engineer if they know who Borland is, they’ll respond “Who?” which typifies the company’s slide into obscurity.

I wish the best of luck to my former colleagues, who I’m sure, will be wondering what is to happen next. I also hope that the new owners, Micro Focus International (who?), have good fortune with their ALM drive. Perhaps the Borland name might live on as a brand for a suite of ALM products – who knows what they’ll do.

In the immortal words of Dr. Seuss “Don’t cry because it’s over. Smile because it happened.”

PHP algorithms: Determining if an IP is within a specific range.

Written By: pgregg - Apr• 27•2009

I spend a lot of time lurking in the #PHP channel (efnet and freenode, please – no flamewars) and this topic is a commonly asked one that usually gets a simplified answer in the form of using strpos(), or at best an ip2long() in a greater than and less than answer.

Unfortunately although people usually understand that an IP address is simply an unsigned 32 bit integer, and is easily determined, usually with $_SERVER['REMOTE_ADDR'], where the real challenge is – is in specifying the range within which they wish to check that IP address.  IP ranges are usually specified in three common ways (in increasing complexity):

  1. Wildcard: 192.168.10.*
  2. Start-End range: 10.1.0.0-10.1.255.255
  3. CIDR*: 172.16.1.0/24

* Classless Inter-Domain Routing

The Wildcard method, or “classy”, allows you to work at Class A (10.*.*.*), Class B (172.16.*.*) or Class C (192.168.10.*) levels of granularity which is how we used to do things in the old days (before the Web decided to make the Internet popular).  But, increasingly, this just isn’t granular enough for practical purposes.

Thus was born CIDR (yes, I’m skipping talking about Start-End ranges for now).  CIDR brought about the concept that we really didn’t need to break networks on 8, 16, 24 bit boundaries and we could be more granular by allowing the use of any number (from 2-30) to specify a range of networks.  Details on why you can’t use “31” is beyond the scope of this article.

CIDR renamed the former Class A, B and C networks as /8, /16 and /24 respectively and reflects the left-most significant bits of the 32-bit IP address.  Thus was born the ability to specify very specific IP ranges in the form a.b.c.d/xx.   However, part of the problem with this is that although it concisely describes the network start and end, most normal mortal humans couldn’t decipher it. CIDR addressing can also be specified in the form of a longer netmask, e.g. a.b.c.d/255.255.255.224

Thus, the simplified form of Start IP – End IP was put in place for mere mortals and is typically used by those without a networking background.  It also features heavily in consumer broadband routers and notably in Microsoft Windows DHCP server.

So having explained how a range, and by inference, that a netmask is, how can we use this knowledge to help us in determining if an IP is within a range?

What this article will attempt to do is guide you though the construction of algorithms to make the checking of IPs simpler.

Logically, Method 1 (the Wildcard), can be easily converted to Method 2 (Start-End range) by using setting Start and End to the Wildcard string and replacing the “*” character with 0 for the Start and 255 for the End, thus for example, “192.168.10.*” becomes “192.168.10.0-192.168.10.255″ which should (I hope) be obvious to everyone.

We can then proceed to evaluate both Method1 and Method2 in the same way.  In this we’re simply going to use the PHP built in function ip2long() on all 3 values and perform a mathematical check for Start <= IP <= End.

list($lower$upper) = explode('-'$range2);
$lower_dec ip2long($lower);
$upper_dec ip2long($upper);
$ip_dec ip2long($ip);
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

We have, however, a complicating factor here – PHP does not do unsigned integers (32 bit) – which would be necessary for this math to work properly.  We can negate this by switing to floating point data types.  PHP stores  floating types as 64 bit and so will have no problem with IPv4 address space (note – this obviously isn’t granular enough for 128bit IPv6 addressing). Therefore the simplest way to solve the Start <= IP <= End problem with IPs and floating point numers is the following piece of code:

$lower_dec = (float)sprintf("%u",ip2long($lower));
$upper_dec = (float)sprintf("%u",ip2long($upper));
$ip_dec = (float)sprintf("%u",ip2long($ip));
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

Next we have the challenge of handing the CIDR netmasks. What we could do is to take a CIDR format IPaddress/netmask and calculate the Start and End IPs of that block and proceed as before – but that would be no fun – and would mean I haven’t really taught anything through this article.

The method we’re going to use here is how all the world’s Internet routers determine if a destination IP is in a specific CIDR address space. And we’re going to get down and dirty with bitmasks and logical bitwise operators.

So using a real world example, my webserver IP 80.76.201.37 and the netblock within which it resides is 80.76.201.32/27, how does this all work?

Well the /27 indicates that the first 27 bits of the IP address are the same network and IP address in that network (range) will have those same identical first 27 bits.  Bits 28-32 are variable and allow 5 bits of variation.  If you know your binary, then this means 32 possible IPs. (However with routing, you can’t use the bottom and top IP from any range as these are special and mean the network and broadcast addresses respectively. [This is also why a /31 isn't much use (except for PPP links) as you can't use the 2 addresses that space gives you]).

So thinking logically, bitwise, if I take my IP address and the CIDR spec, then all I have to do is check that the first 27 bits all match and I’m good. Correct.  So how would we do this in PHP? Sound’s simple, lets just use PHP’s bitwise logical AND operator: &

Again, correct.

In order to do this we need to convert 27 into what 27 really means – a 32 bit number of 27 ones and 5 zeros in binary (which is what 255.255.255.224 really looks like).

In pseudo-code you could then do if (IP & BITMASK) == (RANGE & BITMASK) then all is good and you know that the IP is within the range.

Visualising this using our real IP address (using the very handy unix tool ipcalc):

Address: 80.76.201.37 01010000.01001100.11001001.00100101
Netmask:   255.255.255.224 11111111.11111111.11111111.11100000
Wildcard:  0.0.0.31        00000000.00000000.00000000.00011111
Network:   80.76.201.32/27 01010000.01001100.11001001.00100000
HostMin:   80.76.201.33    01010000.01001100.11001001.00100001
HostMax:   80.76.201.62    01010000.01001100.11001001.00111110
Broadcast: 80.76.201.63    01010000.01001100.11001001.00111111
Hosts/Net: 30

You can see this in the Wildcard line of 0.0.0.31, and the Network ORed with Wildcard results in the Broadcast address: 80.76.201.63.

Knowing this, then the IP address ANDed with the Network address will result in the same value as the Range ANDed with the Network address and so can be used as a comparison for an IP residing within that broadcast range.

How can we work out this Network address in PHP, again we have two strategies, one is to so a simple substr() and take the left most significant bits of the range and then simply pad out to the right with 0s.  Or we can do some math with “NOT of 2 to the power of (32-range) – 1″. Thus for our value /27 this gives us the decimal value 31, NOTed results in (65536-31)  (representational in the bit form – PHP will see it as a negative integer, but we don’t need to worry about that).

I’m sure by now, your screaming for some code (and if you stuck around this long, you really deserve it).

Code to manipulate a range/netmask into a broadcast address, using math, assuming:

$ip "80.76.201.37";
$range "80.76.201.32";
$netmask 27;

We can convert the IPs to long integers using ip2long (denoted by variable_dec – dec being short for decimal):

$range_dec ip2long($range);
$ip_dec ip2long($ip);

This gives us the basis of our math, we now just need to work out the broadcast address.

Strategy 1 using str_pad to create a string by padding with 1s and 0s.

$netmask_dec = bindec( str_pad('', $netmask'1')
                     . str_pad(''32-$netmask'0') );

We can achieve the same result though mathematics by NOTing the wildcard value. This is our Strategy 2:

$wildcard_dec pow(2, (32-$netmask)) - 1;
$netmask_dec = ~ $wildcard_dec;

Once we know the netmask address (in decimal) as we have here, we can know that, if by ANDing this with the original IP to check results against the Range ANDed with the Netmask, then the IP is within the range defined by the range/mask.

This can be checked easily with:

return (($ip_dec $netmask_dec) == ($range_dec $netmask_dec));

 

I have pulled all of this logic together in a easily included file to provide a single function called ip_in_range($ip, $range) in which $ip is the IP address you want to validate and $range is a any of the above formats, Wildcard, Start-End addressing or CIDR.  The function will return a simple TRUE or FALSE if the IP is in that range.

The source code to the all-in function is available here:

http://pgregg.com/projects/php/ip_in_range/ip_in_range.phps

With an example run (and source code):

http://pgregg.com/projects/php/ip_in_range/test.php

I hope this article has been educational, please feel free to leave comments or feedback.
Update: There have been questions about PHP’s signed integers and my use of bit operations in the code.   It is important to recognise that when dealing with signed or unsigned 32 bit integers purely as bit patterns for masking with a netmask or broadcast address pattern – the fact that a number (128.0.0.0 or above) really is negative, doesn’t have any impact on the validity of the result.  The only impact to not having signed 32 bit integers is in the Start-End range check (example 2 above: 10.1.0.0-10.1.255.255) where a range spanning the switch from positive to negative would be catastrophic to the check.  We can safely work around that problem by using floating point numbers as we are only doing <= and >= comparisons and not attempting any bitwise operators (which don’t work on floats).

Caught red-handed.

Written By: pgregg - Apr• 26•2009

Only two hours after he was supposed to be asleep…

7-IMG_7460_crop_500w.jpg

Only light source was the Nintendo DS.

tail -# file, in PHP

Written By: pgregg - Apr• 23•2009

I read Kevin’s Read Line from File article today and thought I would add some sample code I wrote a while back to address a similar task in PHP – How to perform the equivalent of “tail -# filename” in PHP.

A typical task in some applications such as a shoutbox or a simple log file viewer is how to extract the last ‘n’ lines from a (very) large file – efficiently.  There are several approaches one can take to solving the problem (as always in programming there are many ways to skin the proverbial cat) – taking the easy way out and shelling out to run the tail command; or reading the file line by line keeping the last ‘n’ lines in memory with a queue.

However, here I will demonstrate a fairly tuned method of seeking to the end of the file and stepping back to read sufficient lines to the end of the file. If insufficient lines are returned, it incrementally looks back further in the file until it either can look no further, or sufficient lines are returned.

Assumptions need to be made in order to tune the algorithm for the competing challenges of :

  • Reading enough lines in
  • as few reads as possible

My approach to this is to provide an approximation to the average size of a line: $linelength
Multiply by ‘n': $linecount
and we have the byte $offset from the end of the file that we will seek to to begin reading.

A check we need to do right here is that we haven’t offset to before the beginning of the file, and so we have to override $offset to match the file size.

Also as I’m being over-cautious, I’m going to tell it to offset $linecount + 1 – The main reason for this is by seeking to a specific byte location in the file, we would have to be very lucky to land on the first character of a new line – therefore we must perform a fgets() and throw away that result.

Typically, I want it to be able to read ‘n’ lines forward from the offset given, however if that proves insufficient, I’m going to grow the offset by 10% and try again. I also want to make it so that the algorithm is better able to tune itself if we grossly underestimate what $linelength should be.  In order to do this, we’re going to track the string length of each line we do get back and adjust the offset accordingly.

In our example, lets try reading the last 10 lines from Apache’s access_log

So let’s look at the code so far, nothing interesting, we’re just prepping for the interesting stuff:

$linecount  10;  // Number of lines we want to read
$linelength 160// Apache's logs are typically ~200+ chars
// I've set this to < 200 to show the dynamic nature of the algorithm
// offset correction.
$file '/usr/local/apache2/logs/access_log.demo';
$fsize filesize($file);


// check if file is smaller than possible max lines
$offset = ($linecount+1) * $linelength;
if ($offset $fsize$offset $fsize;

Next up we’re going to open the file and using our method of seeking to the end of the file, less our offset, here is the meat of our routine:

$fp fopen($file'r');
if (
$fp === false) exit;

$lines = array(); // array to store the lines we read.

$readloop true;
while(
$readloop) {
// we will finish reading when we have read $linecount lines, or the file
// just doesn’t have $linecount lines

// seek to $offset bytes from the end of the file
fseek($fp$offsetSEEK_END);

  // discard the first line as it won't be a complete line
// unless we're right at the start of the file
if ($offset != $fsizefgets($fp);

// tally of the number of bytes in each line we read
$linesize 0;

// read from here till the end of the file and remember each line
while($line fgets($fp)) {
array_push($lines$line);
$linesize += strlen($line); // total up the char count

// if we’ve been able to get more lines than we need
// lose the first entry in the queue
// Logically we should decrement $linesize too, but if we
// hit the magic number of lines, we are never going to use it
if (count($lines) > $linecountarray_shift($lines);
}

// We have now read all the lines from $offset until the end of the file
if (count($lines) == $linecount) {
// perfect – have enough lines, can exit the loop
$readloop false;
} elseif (
$offset >= $fsize) {
// file is too small – nothing more we can do, we must exit the loop
$readloop false;
} elseif (
count($lines) < $linecount) {
// try again with a bigger offset
$offset intval($offset 1.1);  // increase offset 10%
// but also work out what the offset could be if based on the lines we saw
$offset2 intval($linesize/count($lines) * ($linecount+1));
// and if it is larger, then use that one instead (self-tuning)
if ($offset2 $offset$offset $offset2;
// Also remember we can’t seek back past the start of the file
if ($offset $fsize$offset $fsize;
echo 
‘Trying with a bigger offset: ‘$offset“n”;
    // and reset
$lines = array();
  }
}

// Let’s have a look at the lines we read.
print_r($lines);

At first glance it might seem line overkill for the task, however stepping through the code you can see the expected while loop with fgets() to read each line. The only thing we are doing at this stage is shifting the first line of the $lines array if we happen to read too many lines, and also tallying up how many characters we managed to read for each line.

If we exit the while/fgets loop with the correct number of lines, then all is well, we can exit the main retry loop and we have the result in $lines.

Where the code gets interesting is what we do if we don’t achieve the required number of lines.  The simple fix is to step back by a further 10% by increasing the offset and trying again, but remember we also counted up the number of bytes we read for each line we did get, so we can very simply obtain an average real-file line size by dividing this with the number of lines in our $lines array. This enables us to override the previous offset value to something larger, if indeed we were wildly off in our estimates.

By adjusting our offset value and letting the loop repeat, the routine will try again and repeat until it succeeds or fails gracefully by the file not having sufficient lines, in which case it’ll return what it could get.

For the complete, working, source code please visit:

http://pgregg.com/projects/php/code/tail-10.phps

Sample execution:

http://pgregg.com/projects/php/code/tail-10.php

Comment: Why Firefox is failing in the corporate environment.

Written By: pgregg - Mar• 26•2009

I’ve sat on this article for a number of years, hoping against hope that the Firefox development team would get off their elite self-indulgent asses and realise that, guess what? – the world doesn’t work the way they think it should.

Don’t get me wrong, I love Firefox. I use it daily for nearly all of my web browsing needs, but there is just one little problem – a massive little problem – and that is why I am writing this article.

Most articles on this subject tend to focus on the lack of IT department deployment and management tools for rolling out Firefox, but that isn’t the issue. Really?
So what is it then?

The answer is very, very simple: Firefox does not work on a real-world company Intranet.  There, I said it. 

Really, it doesn’t – the Firefox development team have decided that in their infinite security wisdom that links from one method (e.g. http://intranet) to a local method (e.g. file://server/expense_claim.xls) are so bad that they won’t even put out a warning.

I feel it is bad enough that it doesn’t work, but silently failing without any alert boxes, or an option saying “Yes, I know I’m risking my life, but really, do let me click this link” or putting file://intranet into the trusted domain is the root cause why Firefox will never be accepted as a corporate browser.

IT departments just do not want to deal with the questions “Why doesn’t the link to the document work?”.  The simplest answer for the IT department is “We only support Internet Explorer”.

Any amount of Firefox protestations saying “Oh! but you shouldn’t be running your Intranet like that.” is not going to change the real-world Intranets, and ultimately it keeps pushing Firefox back from acceptance into the Corporate world.

Until the Firefox is able to be used the way that real users want to use it, IT departments will continue to push that reliable old line that we only support IE.

Welcome to the real world.

https://bugzilla.mozilla.org/show_bug.cgi?id=84128
https://bugzilla.mozilla.org/show_bug.cgi?id=122022

TinyURL PHP “flaw” ?

Written By: pgregg - Mar• 19•2009

The Register is running a story today TinyURL, your configs are showing which points out that TinyURL has a /php.php page displaying the contents of phpinfo().

The article then goes on to make some scary sounding claims from security consultant Rafal Los “Why would you want to run a web service as ‘Administrator’ because if
I figure out a way to jack that service, I completely, 100% own that
machine.” and “More importantly… why is this server running as ROOT:WHEEL?!

Sorry Rafal – but you appear to have no idea how web servers work, or all that much about (web) security.

All unix based webservers start as root if they want to bind to the restricted (and default) port 80, after which they switch to the configured UID for request handling.  So, right there, goes all Rafal’s claims about pwning the machine.

Check your own server, the _SERVER and _ENV values will reflect the
starting shell/environment, which just happens to be root.  In
other words, there is nothing wrong with the settings. Having said that, they do have register_globals turned on, which isn’t ideal – but it isn’t a gaping hole if the underlying php code is safely coded.

Also to TinyURL’s credit, they are running Suhosin patch to harden their server.  They’re also running the latest production PHP (which is more than I can say).  Granted, they probably don’t want to be exposing phpinfo() – but this all just an overblown storm in a teacup.