The Essence of Big Data

It seems like the term "Big Data" has come to refer to so many things that it has become more of an aspirational marketing term than a technical one.
In dealing with "big data" projects at work I have come to a definition that seems useful.

Big Data is composed of several systems that work together:

• The hardware that enables all the other algorithms needed to handle and analyze the data of interest.
• The algorithms which efficiently load, transform and store data from varied, high volume data flows.
• The algorithms which retrieve and manage the stored data in a way that allows other algorithms and s/w tools to act on it efficiently.
• Correlation algorithms which automatically comb through the data looking for data items which are related in ways which might be "interesting" to the end user.
• The visualization tools which are used to look at the automatically flagged "interesting" subset of data in order to determine if the data is actually useful and if so, how.
• The actionable information that the end user extracts from the system which he uses to further whatever goals originally justified implementing the big data system.

The hardware, ETL and data storage parts have been addressed by a fairly large number of vendors using proprietary and open source methods. You could say that the data handling platform is becoming a commodity because the movement of data is an undifferentiated requirement for all big data users.

What is still hard is algorithmically filtering the flood of incoming data to pull out the nuggets of interest so that someone can confirm their meaning. The aspects which qualify something as interesting differ from industry to industry and company to company so coming up with a common, turn-key solution may not be possible. So, until that statement is proven false, the seller's market for data scientists will continue.

Mixing Metaphors: BAC! Who left this MES?

I have been learning a bit about building management systems (BMS) and realized that, abstracted the right way, the manufacturing execution systems (MES) of semiconductor fabs are pretty similar in overall concept and components. That parallel made it much easier for me to frame my learning. 

The brains of the system: BMS vs MES.

These s/w systems take schedules, control targets and feedback to make the building "work" (literally).

The communication protocols: BACnet vs SECS/GEM.

Granted, building equipment use many more protocols than just BACnet, some of which are proprietary to the equipment vendor.

 The building equipment which is actually being controlled for the purpose of making the building useful for its owners. 
An office building could be thought of a making an environment conducive to worker productivity with maximum efficiency.
A fab could be thought of as controlling the flow of materials between equipment to maximize output of wafers/chips at minimum cost.

 The sensor systems which allow the control system to make smart choices about controlling the building equipment.
For an office building this might mean feedback control for HVAC (don't over cool) and switching off unused lights or dimming lights for daylight harvesting.
For a fab this might mean monitoring the voltage and flow rates for a particular piece of process equipment and adjusting the process recipe for the next lot or wafer to ensure uniform film properties from lot to lot.

The Essence of a Marketing Requirements Document (MRD)

A few thoughts about writing MRDs.

NOTE: The MRD should describe WHAT needs to happen overall and between parts. The MRD should not (usually) describe HOW all the parts get implemented - that is for the engineering design document.

1. Describe the end-to-end scope of the problem to be solved
2. Break the problem into logical sub-problems
3. Describe the inputs required to resolve each sub-problem. This includes:
• human interfaces for data input 
• one time 
• interactive / iterative 
• machine / data inputs from external data
• machine / data inputs from internal (transient) data
4. Describe what output should be generated by resolving each sub-problem. This includes:
• which data is needed as "the" output. i.e. the "permanent" data. 
• What is the expected input format of the consumer(s)?
• which data is needed to address another sub-problem. i.e. transient data.

All of this should be written with an eye to the system in which the functionality described by the MRD lives.

• Every input is the output of another system, ideally described by an MRD (reference it if you can).
• Other systems may need the output of the system described by your MRD. Include these systems as examples in your MRD to give color to the bigger picture problem being solved.
• Human input interfaces (User Interfaces), Machine Input interfaces (APIs) and Permanent Data stores (HDDs or Databases) may be shared between multiple systems. If they are, or should be, note that explicitly.

One obvious challenge, given the recursive approach to MRD writing given here, is figuring out where to stop.

How big should the scope of THE problem be?

My experience: when in doubt, make the scope too big. Then scale back the scope during reviews based on feedback from the stakeholders.

What is Agile Development an Answer to?

I got to thinking about some of the problems I see at work around software development & roadmap and decided to apply some systems thinking to the situation. This is the result.



The key learning:

• Really good use case validation is probably the largest leverage point.
• Agile development can be an alternative to really good use case validation.

No surprises there but it is interesting to see the dynamics that lead to those conclusions. The feedback loops suggest alternative paths to address the customer acceptance problem when neither use case validation improvements nor agile development are feasible. For example:

• What if you refused to add late features and managed the initial urgency to gain product acceptance? As long as the gaps are fixed in the medium term, the improved roadmap credibility may be enough to gain acceptance in the face of gaps next time because the customer believes your roadmap claims.
• If apps and product managers are failing to validate use cases sufficiently, can you increase scrutiny on requirements by engineering and increase insistence on complete test case details by SQC to minimize factors which cause schedule slips and perhaps offset the slips caused by feature adds?

The Essence of Data Analysis

Cloud Computing, Virtualization, IaaS, PaaS and SaaS - Part 2

In part one of this post, I looked in to the question of why cloud computing is important. In this post, I will look into the question of what cloud computing is.

NIST's definition of cloud computing gives a useful model for deciding what is and is not a cloud deployment:
 "cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction."

But to understand what "the cloud" is really made up of, it helps to look at how cloud computing is packaged and sold.

• Infrastructure as a Service (IaaS)
• Platform as a Service (PaaS)
• Software as a Service (SaaS)

Infrastructure as a Service (Iaas)
In this cloud offering, the seller provisions virtualized compute, network and storage upon which the buyer installs his own operating system and software. This is most easily thought of as buying bare computers and storage onto which your own IT dept will put everything required for your specific uses.
A few major vendors who offer these products are: Amazon Web Services, RackSpace and AT&T Synaptic Hosting.

Virtualization software is key to making IaaS feasible. Without virtualization, physical hardware would need to be provisioned for each customer. This would make it prohibitively expensive to quickly provision, re-provision and scale resources to meet customer needs.
The major virtualization s/w vendors include VMWare (ESX), Microsoft (HyperV) and an open source virtualization solution Xen.

More on virtualization in another post.

Platform as a Service (Paas)
In the platform level cloud offering, the seller provisions some combination of operating system, middleware and runtime packages which enable the buyer to develop and run applications of their choosing. This can be thought of as buying access to services which enable programs to easily access the resources they need to run.
Vendors provide a variety of offerings in this space. A few major offerings are below:

• Google App Engine provides a number of runtime environments for web application developers.
• Amazon Simple Storage Service (S3) provides file based storage to any application which needs it while Amazon Elastic Block Store (EBS) provides block level storage for applications the need direct storage access (like DBs).
• Microsoft Azure provides access to a few runtime envrionments,  SQL DB services for applications which need DB services and the virtual network fabric required to link together multiple server's services.

Software as a Service (Saas)
In this level of cloud offering, the seller provides the buyer with access to end applications while hiding the infrastructure, middleware and runtime components. Pretty much every time you hear about a "web application" being offered by a company, you are seeing a SaaS product.
There are probably thousands of SaaS offerings available so covering any significant fraction of the space here is futile. But here are a few examples of SaaS offerings:

• Web email - from google, Yahoo, etc
• Office 365 and Google Apps - On line productivity apps for creating, accessing and collaborating with others on a variety of document types (word processing, presentations, spreadsheets, picture editing, etc).
• Sustainable Supply Chain (SSC) - Supply chain survey management for corporate social responsibility reporting from CSRware.
• Facebook and Google + - public social networking tools for keeping tabs on your friends around the world.
• Socialcast, Clearvale , Jive and Spigit -  Enterprise social networking tools for keeping abreast of news and status, smoothing workflow, fostering collaboration and spurring idea generation in a corporate context.

One additional consideration is which user base a cloud deployment is intended to serve. This leads to the ideas of: Private, Public and Hybrid Clouds

• Private cloud = services and (often hardware) are strictly for a single company's use. Frequently this implies that the company consuming the services will deploy the cloud services on hardware behind their corporate firewall.
• Public cloud = services are available for all users anywhere. Though users may need to pay for services... This is the context in which most people experience the cloud because of the heavy reliance start-ups offering SaaS products have on public cloud services.
• Hybrid cloud = multiple cloud systems connected in such a way as to allow programs and data to be easily moved between private to public clouds.

Cloud Computing, Virtualization, IaaS, PaaS and SaaS - Part 1

What is cloud computing and why is it important?

On the question of importance, cloud computing addresses several business problems.

Perhaps the most mundane problem that cloud computing addresses is one of cost for getting applications up and keeping them running.

• Availability 
• Because of the underlying technology used in cloud computing (Virtualization - more on this next post.) it becomes relatively easy to implement failover and high availability for servers. This makes the customer experience better (fewer interruptions) and allows more routes to addressing business critical applications (lowers barrier to entry on some enterprise applications).
• Application deployment & maintenance 
• Depending on the level of the cloud you engage at (Infrastructure, Platform or Software - more on this next post), the level of IT expertise required to use the applications / services you want can be greatly reduced. Software patching, upgrades, hardware provisioning and maintenance is left to the cloud providers who have this expertise on staff.

These "mundane" aspects of cloud computing enable significantly less mundane capabilities. The most immediate of which are the new business models that become feasible.

• Software start ups galore
• Because compute power can be purchased on-demand and in relatively small units, capital costs for starting a software company are significantly reduced. This enables more players to try more things in search of the next big thing.
• F2P (Free to Play), Freemiums and ad-driven businesses
• Because the marginal cost of adding users to a centralized s/w application can be nearly zero when deployed in a dynamically scalable infrastructure such as the cloud provides, companies can explore business models that "give away" software, or more precisely: give away access to software, while closely tracking and tailoring the user experience more easily. This leads to businesses that thrive on micro transactions within the software or on ad revenue instead of from sales of the software itself.
• On-line special events
• Relatively low cost and rapid provisioning of compute, network and storage resources allows for companies to generate increased community engagement or media attention by using this burst capacity to host periodic special events. For a small fraction of the cost of buying and setting up the h/w and software required to gain this capacity, the same buzz generating potential can be realized.
• Mobile
• By using the relatively limited compute power of networked mobile devices to drive the user interface and pushing the computationally intensive tasks to servers in the cloud, the capabilities of mobile devices become nearly unlimited. This creates a new class of applications (e.g. voice recognition, navigation, etc) for entrepreneurs to explore and market, driving sales of mobile computing hardware (e.g. smart phones and tablets) or of the applications (as apps or as services) themselves.
• Distributed work force, virtual desktops and collaboration
• Because public cloud resources can be accessed from any internet connected device and data storage is easily centralized, the cloud makes it more feasible than ever to provision specialized applications to anyone, anywhere, to keep company data centralized and provide tools by which physically separated groups can easily interact and exchange information in real time in order to improve productivity.

As Corporate Social Responsibility (CSR) and sustainability concerns increase, the "green" aspects of cloud computing also become more important.

• Economies of scale in "green" data centers may mean that renting capacity via "the cloud" is greener than you can afford on your own.  
• Making an efficient data center requires high up front capital and expertise costs. Considerations like the ones below are more affordable by large businesses or by dedicated cloud hosting companies where the risk associated with the capital costs are more acceptable for the long term return.
• efficient server h/w
• low energy and passive cooling strategies
• occupancy based lighting, energy efficient lighting fixtures and daylighting strategies
• green power purchase agreements
• on-site generation via renewables (e.g. co-gen, bio-fuel based fuel cells, PV, Solar thermal, etc.)

Next time some details about the the "what" part of the question.

Saving the Savages = Breaking the System

I just finished reading The Wayfinders - Why Ancient Wisdom Matters in the Modern World by Wade Davis. Besides being one of the best arguments I've read on why sustainability matters and where we might "relearn" how to achieve it, there was a section on the consequences, however well intended, of misunderstanding the context of a culture and of applying the wrong solution to a problem because of it.

The story is of the people of the Kasuit desert in northern Kenya. For hundreds of years, they lived in this desert. They developed a culture that embraced the difficulties of living in the desert and, therefore, did just fine. Until severe drought, combined with nearby wars brought this region to the attention of the West. At which point, the United Nations took actions to "save" them from themselves.
Following is my summary of Wade's story.

The culture the originally developed worked like this:

To ensure that enough livestock survived severe drought, large herds of livestock were required.
To maintain large herds required large families, stewardship of the land and a nomadic lifestyle (so that the carrying capacity of any one area was not exceeded).
The requirement to have large families resulted in polygamy being common. But to maintain social harmony among the young men in a group where there were not enough women for every man, the culture developed rites around sending young men away to tend the herds. These rites bound together the men involved in them, creating life-long social bonds which reinforced the "commonness" of their use of the land (avoiding the "tragedy of the commons"). These rites also instilled a large amount of prestige in the men, allowing them to accept the situation which supported the large families.

Then the UN intervened and added a few things:

First they added the idea of private ownership of the land and, thereby, permanent settlement. This required the reduction of herd size which quickly led to dependence of outside aid to keep the people alive.
Outside aid included western education. The education broke the traditional rites of passage (by instilling the idea that such a life was "beneath" an educated man), which broke the personal ties which were necessary, among the other factors, to allow life in the harsh environment.

The entire culture collapsed and, along with it, the ability to sustain life in that place.
All with the best intentions...
...but without sufficient respect for the wisdom of the people who had learned to live within the limits of the place they inhabited.

Between a Gladwell and a Strogatz

A bit of an odd juxtaposition but here are two talks about ideas that I find very inspirational for my work.

• One says that seemingly simple things may actually be complex.
• The other says that seemingly complex things may actually be simple.

 But both suggest a deeper order exists somewhere inside the system.

------------------------------------------------------------------------------

 Malcolm Galdwell: On Spaghetti Sauce 

"There is not perfect spaghetti sauce. Only perfect spaghetti sauces"
"We cannot always explain what we want deep down."
"What is the great revolution in science over the last 10, 15 years? It is the movement from the search for universals to the understanding of variability."



-----------------------------------------------------------------------------

Steven Strogatz: On sync

Simple rules give rise to complex behaviors.
Feedback produces surprising behavior that we don't fully understand (we can say "how" (simulate) but not "why").
... and the Book is better than this video but the video does give some sense of the main idea.