Inmon gets it wrong (again...)

{Prelude: I’d like to tip my hat to the incredible hubris of a man who says in his bio that he’s “universally recognised” as the “father of data warehousing”. At least have the decency to get someone else to say it and then use it as a quote.}

In his article last week (A Data Warehousing Pop Quiz) takes aim at the new 24x7 / Real-Time data warehousing vendors (and pundits). His “beef” with them is that data warehouses are “for management” and they “don’t need” data this frequently. He then goes on to extrapolate that real-time data warehouses are “not data warehouses” they’re actually “transaction systems”. I call bullsh!t.

Two key trends in business over the last decade have brought real-time data to the forefront: 1-pushing decisions down closer to the “coal face”, 2-spreading work across many time zones world wide. Added to that the number of systems in a business has mushroomed from 2-3 in the 80’s to 50-100 today.

The data warehouse has become the natural point of integration for many of these systems. It is precisely because it is “non-volatile and time invariant” that it is an ideal integration point. Moreover it is also much easier to deliver accurate data in a single place.

Finally, here is my key point, it is entirely possible to architect a ”non-volatile and time invariant” data warehouse that is also available 24/7 with near real time updating. In fact, it is the only way to make 24/7 work reliably. 

If you are a Telco looking to implement a real-time data warehouse then I would like to introduce you to my employer: Scorecard Systems. We can implement our Subscriber Analysis application (on your current database platform) within 90 days.

…with near real-time updating, perfectly balancing values and full auditing. 

Sync *any* Nokia phone with iSync in 5 easy steps

OK here’s the scene… You just got your brand new Nokia. You’ve dutifully charged it fully and now your ready to play. First thing you do is pair it up to your Mac to sync your contacts onto the phone. And then it happens…

“iSync does not support this phone.”
WTF! Fear not, help is at hand and I’m going to save you hours of googling.

If you have any recent Series 40 (or Series 60) Nokia then you definitely can sync your phone with iSync! All it takes is a little bit of text editing.

Step0: Quit iSync!

Step1: Check whether Nokia offers a plug-in here: http://www.nokia.com/mac/isync. If your phone is here then you are a lucky son of a gun! Otherwise download the plug-in for the most recent similar model. For example my phone is a 6600 fold and I downloaded the plug-in for the 5700.

Tip: You need to download the plug-in for a phone that uses the same ‘series’ of software either Series 40 or 60. N an E phones are generally Series 60, others are generally Series 40. Check the Nokia site to be sure.

Step2: The download is a ‘.dmg’ disk image which will mount (mine was ‘Nokia-5700-iSync_Plugin’). This will contain a ‘.pkg’ package file. Right click it and select ‘Show Package Contents’. Now double click on ‘archive.pax.gz’. This will create some new files in Downloads/Library/PhonePlugins. Copy the ‘Nokia-XXXX.phoneplugin’ folder to {username}/Library/PhonePlugins (create this folder if needed).

Step3: You need to replace all references to the other phone model with your phones model in 2 files: Info.plist and MetaClasses.plist. Open each file in TextEdit and simply replace 5700 with 6600. Save the files and close them.

Step4: IMPORTANT! This is the step that got me. iSync won’t see your phone until you add it to the *master* MetaClasses file. In Finder, go to Applications, right click on iSync and select ’Show Package Contents’. Go to Contents/PlugIns/ApplePhoneConduit.syncdevice/Contents/PlugIns/PhoneModelsSync.phoneplugin /Contents/Resources and edit MetaClasses.plist (make a backup copy!). Find a phone that is similar to yours (same series, similar age - I searched for ‘Series40’) and copy the entire entry for that phone starting ‘<key>Nokia…’ and ending ‘</dict>’ replace all references to the other phone with your phone model. Save the file.

Step5: Start iSync, add device and *voila* you’re done. Syncing heaven! Good luck. I hope this works for you too.

Hat Tips: The following gave me many breadcrumbs on the way to getting this working: James Lloyd, Michel Legnered, Otmar (last comment) and Mac OSX Hints

Term Bankruptcy: "New Paradigm"

Term Bankruptcy: occurs when useful terms become so abused by marketing, mis-information and spin that they are no longer meaningful.

I hereby declare the term “New Paradigm” to be bankrupt of all meaning.

So stop using it. This new thing you’re doing might be great but don’t call it a “new paradgim” in whatever because it just doesn’t mean anything and no one believes you.

Goodbye Flickr, you suck!

You have got to be kidding me! I just got the email below from Flickr, the much blogged about photo-sharing site which is owned by Yahoo:

Subject: [Flickr] You are near a limit of your free Flickr account

Hi xxxxxxxxxx,

You have 172 photos stored on Flickr. Once you hit 200,
you’ll need to upgrade to a Flickr Pro account or you’ll
only be able to see your most recent 200 photos. Nothing
will be deleted, and if you upgrade, you’ll have unlimited
space for all your things.

Perhaps you’d like to purchase a Flickr pro account? Its
unlimited and you get video and stats too!

You’ll even get 3 months free for purchasing before 30
September 2008!
…
The Flickreenos

The last time I got a message like this was from Hotmail and, hey presto, I am now a Gmail user. {Hotmail, you suck too for deleting my emails without warning…}

I honestly can’t believe that Yahoo are trying to pimp me for $25 to upgrade to a “Pro” account just to get more than 200 photos. It’s like a flashback to 1998. In the future I’ll be sharing photos on:

• Google Picasa (amazing desktop software, online site is OK)
• Photobox (unlimited storage, lots of printing and gift options)

Note to Yahoo: Are you guys crazy? My time and attention have got to be worth more than $25. Surely you want to be the place where I store my photos and send all my family and friends to view them. If you’re desperate for cash you should add some easy printing and gift services like Photobox. Goodbye.

Is the inflation rate manipulated?

Hat tip to: Junk Charts for pointing me to this in the post “Proof of rampant U.S. deflation”. (Title is tongue-in-cheek)

Have a look at this very interesting article from the San Diego (California) Union-Tribune, “The Fed’s inflation gauge isn’t realistic”. The basic point is that the US government has systematically revised the inflation calculation downward since the days of late ’70s “stagflation”. Here’s the chart…

This paragraph really jumped out at me:

“These price rises have been particularly hurtful to people living on fixed incomes – including the elderly – since their government-provided benefits are lagging behind the rising costs. Social Security and other government benefits are tied to the official inflation rate.”

I’m guessing that the figures were revised to prevent government spending on social welfare programs (US Social Security, Pensions, etc.) from spiralling out of control. Can you imagine the effect on the US deficit of raising social security payments by 11% per year?

The ’90s brought a seemingly new era of transparency in these matters whereby the government explicitly linked it’s policies to “key indicators”. Call me naive if you will, but I certainly did not realise that they were then manipulating these very indicators.

Total Surveillance - A new legal framework

In the previous post I proposed that total surveillance is already a reality. Here I’m suggesting a new legal framework to manage it.

Currently there are hundreds of seperate databases and archives managed by dozens of regional forces and governement agencies. The techiques used to search this information are just as varied.

Some of these require warrants for access however these are granted in secret and individuals cannot find out about them, even under the Freedom of Information Act. There is currently no record of what data was gathered under the authority of each warrant (AFAIK).

There is also no legal requirement to use warrant data for the warranted crime only. In other words, if they suspect you of terrorism and listen to your calls you could actually end up in jail for tax evasion using the same data.

The problem of privacy is foremost in my mind. I would describe myself as a compassionate libertarian. I hate the idea of total surveillance but I don’t think it can be avoided because voters are more scared of terrorists than the loss of liberty. Sad (and deeply wrong) but true.

So, how do we get the benefits of catching terrorists without allowing the government to send fines to parents who’s teenagers use BitTorrent?

My suggestion: we need a centralised magistrates court for ALL surveillance requests.

In order to access data, investigators must submit an overview of the person they are looking at, the criminal activity suspected, the data they will search and the length time they will require access.A (big) team of experts should carry out the searches and supply the results.

Search results would be registered with the names of all the parties referred to in the data returned. The magistrate court will need to operate in secret, however the magistrates would be tasked with ensuring that surveillance requests were valid, targetted at a real crime and not “fishing expeditions”.

In particular, the police must be prevented from doing blanket searches except for the most serious crimes such as terrosim, murder, etc. They should be required to clearly state what they want to find and how they expect to find it. The crown prosecution would be limited to using the data as evidence only for the crime that was listed in the search request.

Also, everyone would be allowed to file a request to see any searches which referred to them and the data returned, excluding ongoing cases. These search requests should be public data and fully searchable by the police (i.e. searching the searches).

This would be a huge improvement on the current situation.

Welcome to total surveillance

A centralised surveillance database could be an improvement on what we have currently in the UK.

The UK goverment recently floated plans for a nationwide surveillance “database” containing all phone calls, emails, web history, etc. Supposed to help tackle terrorism, crime and other Really Bad Things.

Of course there’s been an outcry that we’re “sleepwalking into a surveillance society.” Everything you do that involves any digital element is already recorded (Calls, Emails, TV viewing, etc). Every step in any urban setting is captured on a CCTV camera. Our motorways have cameras on almost every mile. Passive surveillance is even more pervasive because digital systems are designed to log everything that ever happens.

The truth is this: we are already under almost total surveillance. It’s not all joined up at the moment but you are under surveillance. It’s just as simple as that.

We’re currently protected by several practical realities. First, it’s very time consuming and expensive to use surveillance. Did you even wonder why no one intervenes in the city centre assualts you see on TV? It’s because no one is actually watching the screens, ever! Second, it’s very difficult and expensive to store and analyse large volumes of surveillance data.

However, it’s still very early days. New techniques and technologies are released daily that make this ever easier. Number plate reading cameras, facial recognition, passive full body scanners, natural language processing programmers, etc. The technical barrier gets ever lower.

So I think we must accept that everything is recorded and come up with a sensible legal framework for the future of privacy. This issue will not go away. We need to think long and hard about how deal with it, now and in the future.

Next up, I’ll look at the elements of a workable system (technical and ethical)…

Business Idea: P2P Backup System (using Git)

In the hope that someone, somewhere will make it happen.

I think there is a fantastic business opportunity for a peer-to-peer (P2P) backup system that uses Git to hash files, split them into pieces and distribute them over BitTorrent to a number of hosts.

Concept:

• Users trade some disk space and bandwidth for free offsite backups.

Details:

• Leverage GIT version control system:
  • Uniquely identify all files using SHA1 hash algorithm.
  • I estimate that 90%+ of PC users data is not unique to them:
    • i.e. Programs; OS; Downloaded data, music and movies;
  • All text data is compressed before hashing to minimise transfer.
    
  • “K safety” concept used to insure that a lost user doesn’t compromise system.
    
    • Could actively ensure that a given number of copies exist.
      
• Leverage BitTorrent
  • Data exchange between users is done over BitTorrent.
  • Each piece of a file is uniquely identified across all users.
  • Pieces can be shared among users with minimal duplication.
  • Central service to maintain details of piece locations and hosts.

Profit Opportunity

• System gains insight into a huge number of files, anonymously. This insight is potentially very valuable for demographic analysis.
• Business is not required to maintain a huge storage farm.

Weak Signal: The 2nd Laptop

Weak Signals - new trends that haven’t reached mainstream adoption.

2 laptops on 1 desk.

This is a new phenomena that I’m seeing more and more in businesses. The 2nd laptop is not provided by the business, but it is the place where the real work is getting done.

Not long ago, if you had a second laptop on your desk you would have gotten some strange looks from co-workers and hard questions from the IT department. However, the explosion of outsourcing and professional services companies (IBM, Accenture, etc.) means that it’s commonplace for people working in a business to be employed by someone else. These workers need “secure” access to the mother ship services which usually means a 2nd laptop.

Thus a new precedent is set that 2nd laptops are not a problem. Now “normal” employees are bringing in their own laptops and keeping them open on the desk all day. Many are using them to do the bulk of their work and simply sync’ing their output back to their “main” machine by email or a USB key.

They no longer have to bang their heads against IT’s glacially slow pace (Windows 2000 anyone?) and obessive Windows-only stance. Most run Vista, which is about 3 years from adoption in your average company, bu plenty are running OSX or Linux.

They aren’t allowed on the corporate network so they’re using WiFi (in cities) or a 3G/EVDO USB dongle (in the burbs). Suddenly all of the company blocks on Facebook, MySpace, webmail and “personal storage” go away.  Webwasher be damned.

The real trend: IT are losing control of the user estate. A day is coming where knowledge works provide their own laptops and all corporate services are web-based. It would certainly be cheaper and easier for all concerned.

Internet Age: Vapid not Stupid

There’s a new meme about: “The Internet is making us stupid (oh no!)”. Nick Carr ostensibly started the hand wringing, although he may have picked up the sickness from The Economist.

The idea (to paraphrase): “The Internet (or Google or whatever) is making us stupid by presenting the obvious answers so easily that we never really learn anything or look any deeper.” We’re all just snackers at the information buffet. We never eat a whole meal and no one cooks.

This seems patently silly to me. There is more information on the Internet than anyone could ever possibly consume. But I sure as hell don’t want to have any less.

The real problem, if I may say so, is recency bias. The Internet is totally focused on now. If now is not available how about yesterday. No? What’s the freshest thing you can give me?

It is almost impossible to trace the origin of an idea in time using Google (or any other search engine). However, no one seems to care about it. No choruses of doom about “the death of attribution”.

We’re addicted to recency. Freshness equals value.

In the Internet Age we’ve become vapid, not stupid.