Manufacturing automation shows us the future of big data in most companies

Here I wrote about big data software techniques as an analogy to manufacturing automation, and then in practice:
http://onelesscut.blogspot.co.uk/2013/01/big-data-software-techniques-in.html

The analogy is perhaps more interesting than the practice. What do robots bring to manufacturing and how does the analogy with big data software techniques play out in the future?

Regardless of repetition, robots bring :

• Accuracy and quality - they execute repetitive jobs to a high standard with repeatable results
• Speed and efficiency - they're great at crunching through repetitive tasks
• Reliability - they're 'always on'
• Low cost - see Reliability; also they can replace people (hey, it's true) 

They also allow integration up the design and manufacturing process, with the encoding of a physical form into a digital representation with CAD/CAM.

Robots are getting more complex. They're getting smaller, cheaper and more autonomous. They can handle jobs that perhaps required human intervention a few years ago. Most of the advances in robotics appear to be down to advances in software, e.g., signal processing, logic, or whatever.

Big data software techniques are like our robots. They bring:

• Accuracy and quality - algorithms manage distribution and execution of repetitive jobs to a high standard with repeatable results
• Speed and efficiency - they're great at crunching through repetitive tasks
• Reliability - distributed processes give greater resilience but also if you get the algorithm right once, it can be applied to truly massive datasets
• Low cost - you can do an awful lot with less (smaller, cheaper) computing power, and for some tasks they can replace people manually sifting unstructured data.

Where now?

Well, first off I think we may see Data Scientists being moved off the big data frontline, away from the data itself and back towards widely applicable algorithms. Scientists are usually first in to new areas of learning but they're quickly supplanted by engineers. As it was with robotics.

Once the (software) engineers have got these techniques working for industry, their role will move to be supportive, with end-users taking the lead. As CAD/CAM is a way for a subject matter expert to apply their knowledge of the physical domain so as to optimise manufacturing capability, so big data software techniques will allow subject matter experts to apply their algorithms to improve a process - sales, production or whatever.

That sounds a bit like marketing flannel, so here are some examples:

1. "Hey computer, I'm worried about benzene contamination in my product. Should I be?"

[Computer starts complex, distributed log analysis of a few millions lines of real-time data.]

2. "Hey computer, find out how much product we've had to flare off and how much it cost."

[In the future, everyone says 'Hey, computer'. Computer finds all the flaring incidents, exactly how much product was sent to flare from where, the value of each product.]

3. "Hey computer, can you reduce my electricity bill?"

[Computer looks at efficiency of every component in a process, tries to optimise usage taking into account the effect on other parts of the process.]

This is elegant as it doesn't need a multi-petabyte dataset for these techniques to show value; it's about using the existing data, reforming it and translating it into new forms on the fly through algorithms.

Ultimately this feeds right back up to the design process for new facilities, processes and even businesses. As Google extends each of our knowledge and even memories, we'll rely on algorithms chewing through lots of data to be our enterprise memory.

Big data software techniques in automation - as an analogy & in use

I really liked this post (HT @stewarttownsend) :

http://gigaom.com/data/why-big-data-might-be-more-about-automation-than-insights/

As we (@sabisu) do a lot of work in the process industries (oil & gas, chems, manufacturing) the analogy of 'big data' techniques to robotic manufacturing processes really worked for me.

What do robots do in manufacturing? We give them small tasks which require relentless repetition and accuracy. Robots don't do anything you couldn't do by hand but they're many times faster, more reliable and less expensive.

That's a pretty good analogy with many of the big data software tools which involve the distribution of work to many processing nodes. This work is repetitive, requires accuracy, speed, reliability and low cost (in all senses). It's a good fit. For example, the 'reduce' part of MapReduce is in fact a software robot, executing an algorithm. For Hadoop clusters, read any redundant architecture in your manufacturing process.

It's easy to see this translating to the real world. Need to aggregate all the flaring incidents at your petrochems plant? MapReduce will do what your Climate Change Manager might spend hours collating.

Need resilience in data acquisition from real-time manufacturing systems? Plenty of options.

Need someone to check all the logs for incidences of benzene contamination 1.2 standard deviations over the mean? Get an algo to do it.

Now, in fact I think most 'big data' technology is needlessly expensive and perhaps sub-optimal for some of these use-cases but the analogy holds.

Top films of 2012

Every year Mark Kermode gives his top 10 films of 2012...well, top 12 in this case...and with a few other honourable mentions thrown in.

Here's a full list to solve those 'what shall we watch tonight' conversations...

You've Been Trumped

Holy Motors

The Raid

(Argo)

Beasts of the Southern Wild

Martha Marcy May Marlene

(Liberal Arts)

Life of Pi

Even the Rain

(Angel's Share)

The Dark Knight Rises

Amour

Skyfall

(The Grey)

(Moonrise Kingdom)

A Royal Affair

(The Hunt)

Berberian Sound Studio

I'm also going to squeeze in Last Shop Standing if I can - about the decline of independent record shops in the UK.

Measured responses via gun control, philosophy and Gary Barlow

There have been few blog posts worth reading in the aftermath of the Sandy Hook horror. Here are some that are linked to each other and other, similar massacres:

http://m.theatlantic.com/technology/archive/2012/07/the-philosophy-of-the-technology-of-the-gun/260220/

http://www.wired.com/wiredscience/2012/07/batman-movies-dont-kill-but-theyre-friendly-to-the-concept/

http://blogs.plos.org/neuroanthropology/2012/07/24/inside-the-minds-of-mass-killers/

Particularly with reference to the first above, my interpretation of these blogs is basically that (you+gun)<>you + gun, i.e., when you pick up the gun, you are a different entity than you were, the gun is a different entity than it was, and the two of you together are a different entity than simply adding you and the gun.

To me, this is beautiful and logical.
Nowhere have I seen this logic applied, where allowance is made for the fact that combining a human with another entity, organic or not, causes them to become non-you, i.e., not the original entity.

Laws certainly don’t account for the fact that (you+gun)<>you + gun. In fact, I'd argue that they effectively disregard the gun altogether, setting parameters only on the basis of your actions. You can imagine the defence, "The gun made me do it", would land you perhaps in a mental hospital instead of prison but otherwise would be pointless.

Similarly, our behaviour changes when we become you+car; it's easy to dehumanise others when they're disguised in a metal box. In fact, when we label another driver based on their car ('typical BMW driver') we effectively negate 'you' out of the equation altogether, leaving just a metal box. And where's the harm in being angry at a metal box?

Everyone drives a car differently to the way they ride a bike. When it's you+bike, the sum total of that entity has a different attitude to risk and pattern of behaviour than you+car, or just plain old you. Regarding ourselves as fluid entities that become changed when combined with different entities (organic or not) allows us to reconsider our actions and reactions. 

The sooner we all realise that our edges aren't lines, that they're blurry boundaries, the better. A jazz pianist friend of mine once talked about being careful of what he listened to, because everything you hear finds it's way into your fingers. He's right - we're porous, badly insulated beings; skin, brains, emotions, everything. We absorb everything and acknowledge consciously a small part of it.

(This is still my argument for not watching Eastenders, listening to Gary Barlow/One Direction.)

Osmosis doesn't judge bad from good. It's a great asset and continual risk. 

I was going to sign off by saying that we all have a responsibility to control what we expose ourselves to but that's palpable nonsense; you can't control everything around you. And perhaps seeking to do so is a mistake, closing doors and limiting options. 

Perhaps all you can do is control how you react to the world, look for the wider implications and be mindful of the impact on others.

Some challenges for 2013

Here are a few sportives I fancy for this year. Some challenging, some not so.

Really they're building up to the Coast to Coast In A Day and Etape Cymru - I think they'll be really tough.

There's a little space in there for this year's Ride With Brad too...

17th February - Cheshire Mini Sportive [booked]

http://cheshireminisportive.blogspot.co.uk/

10th March - Jodrell Bank Classic

https://www.sientries.co.uk/event.php?event_id=859

24th March - Wiggle Cheshire Cat

http://www.kilotogo.com/index.php?option=event_detail&event_id=49

21st April - Manchester-Chester-Manchester

http://manchester-chester-manchester2013.blogspot.co.uk/

18th May - Keswick Sportive

http://www.highterrainevents.co.uk/11.html

29th June - Coast to Coast in a Day [booked]

http://www.opencycling.com/

14th July - Evans Peaks Ride-it

http://www.evanscycles.com/products/evans-cycles/ride-it-peak-district-sportive-road-entry-july-2013-ec043942

4th August - Ride London [in the ballot]

http://www.ridelondon.co.uk/

8th Sept - Etape Cymru [booked]

http://www.etape-cymru.co.uk/

There are a couple of cyclocross events I'm thinking of at the start of 2013. And a couple of other sportives I may yet plump for...I feel I need more hill practice...

Whilst I've got the turbo-trainer...and the bikes...I haven't had the time so far, so I'm at a base level of fitness somewhere around 'lardy'.

Product/Market Fit and Qualitative Value Assessment in Enterprise Software

I'm getting obsessive about product/market fit. That's probably a good thing and there are some great posts out there that help you get started with the concept.

Andreessen famously made the point that when a start-up has a good product/market fit, the product is virtually 'pulled' out of the start-up by the market. Oh yes, that sounds great doesn't it? Struggling to meet demand is every start-up's dream.

I think that works really well in the B2C sector but not so well in the B2B sector, where a new software product needs to get through numerous layers of shitty bureaucracy at all stages; it's hard to generate awareness in a market cluttered by also rans with big PR budgets; IT people are sometimes not the most accommodating of new ideas; procurement departments enjoy playing with deals; big enterprises enjoy playing with little ones (in every sense).

What trumps all these issues is Value - hence, the often meaningless phrase, 'Value Proposition'. If you can point to a reliable, business (not IT) driven, hard ROI expressed in £/$, then you can jump the hurdles. Hence, I'm interested in the link between Product/Market Fit and Value.

Here's my theory. Let's start by saying that...

Customer Value is proportionate to Usable Product Functionality

Ok, so the more your product can do, the more value a customer can get out of it, with the proviso that it has to be useful to be valuable.

But Usable Product Functionality is basically describing a Product that's a good fit for a particular set of requirements, i.e., a good fit for a particular Market. Andreessen doesn't really talk about the problem that drives the Product in his seminal blog-post, but to me the problem is the key component of the Market. After all, you build a Product in response to a Market need. 

In terms of an individual Customer Value proposition the Market is limited by the size of the enterprise network. It might be equal to the number of users within an enterprise - or, as with Sabisu, it could include users outside the enterprise. The Customer Value is dictated by the Value Potential of the problem the product solves; having a genuine high value problem to solve is the key. It might be a high value problem for a few users, or a low value problem for many users. This 'genuine high value problem' is analogous to the 'must have signal'.

So within a single Customer, Value Potential is simply:

Value Potential = Value of Problem * Number of users 

This means that all the other B2C Product/Market fit characteristics that you'd like to see apply to B2B software; virality drives the 'Number of users', new products can still create new Markets or define problems that have not previously been addressed.   

Let's make this pretty obvious statement:

Customer Value is proportionate to Value Potential

But it isn't just that. That 'good fit for a problem', or 'value potential' needs to be accessible. Every proportional equation needs a constant (remember kids that y=kx) so how about:

Customer Value = Efficiency * Value Potential

Where Efficiency is the ease of extracting value from the solution. You could see it as related to effort required to realise value (so, k=1/effort required) which in enterprise-software-world usually means technical services. Or you could say a solution that's easy to use is easier to extract value from (k=ease of use).

Customer Value = Ease of Use * (1/Services Required) * (Value of Problem * Number of users)

[Here's an interesting game; pick an ERP software implementation of your choice and run it through the above. Qualitatively it's not great, eh?]

Customer Value and Vendor Value really align here. Efficiency is very important to both parties as it limits the Vendor's ability to meet customer needs when the right problem has been identified, e.g., an inefficient platform drives lots of efficiency limiting behaviour such as support calls. 

Note that the contribution of the product itself to the analysis is limited to 'Ease of use'. Everything else is irrelevant so long as it's solving a valuable problem. Just as with the Product/Market Fit concept, the product just has to basically work. It has to be viable - a minimum viable product.

Product/Market Fit is traditionally focused on Vendor Value, i.e., do I pivot because the product/market fit isn't right, or persevere because I believe it is? (cf. Lean Startup). 

The Product/Market Fit calculation still works because Market Value is related to the aggregate of all those individual enterprise problems - that aggregate effectively drives the what the vendor can extract from the current market. If the Value Potential isn't there then a pivot is needed.

So, Vendor Value Potential is proportionate to Sum for customers(Value of Problem * Number of users)

Of course, there's no problem having stacks of Vendor Value Potential if you can't exploit it. If it's easy to exploit, it's efficient, so let's bring that back in again to complete our qualitative equation. 

As a Vendor, Services aren't necessarily negative - in fact they're irrelevant so long as they don't affect your likelihood of a sale. Ease of Use is still relevant as it drives down the cost of maintaining a customer.

What is negative is a high cost of customer acquisition, perhaps because the Market is hard to reach, or it's a new class of product with limited awareness that therefore requires lots of education before the tills can ring. 

Also as we're looking at the Market as a group of Customers, we need to account for the 'Probability of Sale'. Now, I'd be the first to admit that this is hard to determine - there are so many factors that affect whether a sale occurs; 

• Lower price than competitors
• Better marketing/sales collateral
• Better case histories/company history
• Better knowledge of the market

For me the Probability of Sale is just that; a value between 0% and 100% which indicates whether, all other things being equal, your solution would be the one chosen. One thing is for certain; the Product isn't going to affect this - your communications are. :

Vendor Value = Ease of Use * (Probability of Sale/Customer Acquisition Cost) * Sum for all customers(Value of Problem*Number of users)

Of course, the quantitive Vendor Value is impossible to ascertain; you'd need licensing, services and reliable 'Value of Problem' and 'Number of users' terms. However as a qualitative guide...it might have some use.

Questions:

• What other negative/positive factors need to be taken into consideration, particularly around the product?
• What else might inflate Customer Acquisition cost?
• Could the Vendor Value calculation be applied quantitatively to a particular prospect, defining whether it was worth pursuing?

Why Social Media has put Albert Camus & me off football

Football is, at best, a trivial game. This is obvious to me but it wasn't always so.

Until my first football game I had little appreciation for team sports. Perhaps that's why the team ethic at work is important to me...perhaps like Albert Camus, French philosopher, substantial parts of my education is owed to football - or, as he put it:

"what I know most surely about morality and the duty of man I owe to sport"

As Wikipedia states, "Camus was referring to a sort of simplistic morality he wrote about in his early essays, the principle of sticking up for your friends, of valuing bravery and fair-play."

Yes, precisely that. 

However, for me football is dying. It has been for a while but it's taken Facebook and Twitter to demonstrate it.

Let's start with Twitter. Everyone knows that Match of the Day is terrible. The pundits are overpaid and  incapable of basic communication. Their insights are insipid and unremarkable to start with, perhaps due to a preoccupation with employing ex-footballers, so they suffer terribly from being reheated every week. Sky is a bit better but not a great deal. As a result the focus shifts away from analysis to comment - at which point, football is lost.

Because that's where Twitter takes off; comments drive more comments but very little analysis or insight. The coverage often gets an almighty kicking on Twitter. As do the players. And the officials. (Amusingly, when referee Chris Foy was confused with cyclist Chris Hoy.)

The Twitter action demonstrates that the football itself isn't the focus anymore. It could be any sport. In fact it could be anything. Wound up fans wind each other up more, as they do on non-football or non-sport messageboards the world over. Fans even wind up their own clubs - which demonstrates the sport's obsession with the media, but also shows how the unique qualities of football that made it important to Camus have been lost in the barrage of irrelevant commentary. Sure, there are pearls of wisdom in there somewhere but they're cast before swine.

Perhaps more serious is what I see in Facebook. Inevitably feelings run high after a match and everyone piles in with vitriolic comments which always look more serious in print, without facial expressions to mitigate delivery, or the chance to buy a reconciliatory pint. People defend the indefensible (cf. recent high profile racist abuse cases) and it all gets a bit nasty. 

These days I watch football at home. Really, there's nothing at the match I need to see, though there's much to be missed. Most importantly, the TV goes off when the half-cocked opinions come out. I avoid the Facebook threads and trending Twitter topics where football is driving people mad. 

Perhaps social media is just exposing what had been there all along. Idiot footballers with little to say have historically been in luck as they weren't required and had limited platforms for saying what little they did. Now we all get to listen to them. 

And everybody else.