I have been learning a bit about building management systems (BMS) and realized that, abstracted the right way, the manufacturing execution systems (MES) of semiconductor fabs are pretty similar in overall concept and components. That parallel made it much easier for me to frame my learning.
The brains of the system: BMS vs MES.
These s/w systems take schedules, control targets and feedback to make the building "work" (literally).
Granted, building equipment use many more protocols than just BACnet, some of which are proprietary to the equipment vendor.
The building equipment which is actually being controlled for the purpose of making the building useful for its owners.
An office building could be thought of a making an environment conducive to worker productivity with maximum efficiency.
A fab could be thought of as controlling the flow of materials between equipment to maximize output of wafers/chips at minimum cost.
The sensor systems which allow the control system to make smart choices about controlling the building equipment.
For an office building this might mean feedback control for HVAC (don't over cool) and switching off unused lights or dimming lights for daylight harvesting.
For a fab this might mean monitoring the voltage and flow rates for a particular piece of process equipment and adjusting the process recipe for the next lot or wafer to ensure uniform film properties from lot to lot.
More specifically, they each help to reduce the friction between all of the things that must be in place to get something that you want.
International Paper - What does it take to get the paper in your printer?
Rights to log trees. The labor to log them. The tools to cut down a tree. The knowledge to use those tools. The transports to move the trees. The machines to pulp the tree. The labor and knowledge to use the pulping equipment. The chemicals to process the pulp into paper... you get the idea...
United Airlines - What does it take to move yourself to Japan?
The money to buy an airplane. The knowledge to fly it. The contacts required and hours spent to negotiate the rights to take off from SFO and land at NRT. The labor and knowledge to service the aircraft... etc...
Google - What does it take to find out about everything on the internet?
The knowledge to create an algorithm that is helpful at finding what you want amid tons of stuff you don't. The programming skills to implement it. The knowledge to build the IT infrastructure to process and store all of the data required to run the algorithm. The servers and real-estate required to hold the servers... how easy would those be to get on your own?
Facebook - What does it take to find all of your long lost high school friends?
The hours and hours of phone calls to numbers in your old day runner (they still make these?) hoping that their parents still remember you and still live there. Or trawling through phone directories looking for the right Joe Smith... ugh...
OR build your own content site which will attract half of the planet AND get them to list their high school... pretty simple...
The remaining companies or topics flip the equation a bit as they are more general tools for reducing friction towards the end of doing something else.
Twitter - How could I publish my thoughts to "everyone" at a reasonable cost?
I could never mail a letter, call by phone or place enough radio and TV ads to do this. What would it cost to generate the lead list and qualify the leads to do this in a more focused way?
Amazon EC2 - How do I start a s/w business that scales without major capital outlays?
How else can I get enough computers to scale my SaaS business to profitability without the friction of convincing someone to front a significant amount of money to purchase and administer a server farm?
Kickstarter - How do I find funding to raise capital to do something people want to see done?
Am I lucky enough to be born rich? Did I get lucky enough to know powerful, rich people? Am I a good enough social engineer to find these people? Do know the right VCs? Is my product profitable enough to a VC for them to consider? What would it cost to build the audience of millions who are engaged enough to put money on the table - sight unseen?
3D Printing - How do I make a complicated, custom physical part in low volume (qty 1)?
The money and space to buy a CNC machine plus the experience and knowledge to operate it? Or the hours spent to find a machine shop that will do a low volume run, now, for a reasonable price?
Genies - How do I do anything with anyone, anywhere at any time?
You have 3 wishes...
The interesting thing about removing the friction around doing "something else" is that it enables new ways for people to do things for themselves and, ultimately, find others who might want those things. Which they then might trade something for (like money). Which sounds sort of like an economy.
Take that to its logical conclusion where friction is, genie-like, reduced to near zero between all people and the resources / skills they hold and what is the purpose of a corporation as we know it today? We could do anything for ourselves by finding and coordinating the right people.
Maybe this does not happen in my lifetime, but the idea of friction seems like a powerful filter for looking at the value of any product or service that you are trying to create today. If it is not reducing friction then you're heading the wrong way.
After a half day of talking with IBM reps about Big Data products and some use cases, here is how I summarized how the pieces fit together.
At time 0 you collect everything and analyze it for correlations to determine which data items are valuable and how they relate to each other (Big Insights platform). Then you build a control model.
Learning from time 0 is used to configure a "real time" strategy for the data analysis and system control.
Streams provide real time processing of data "on the wire" - nothing need be stored. The output of this is three fold:
"Live" reports for users
A data subset to feed to the data warehouse
Control signals to feed back to the data collectors to adjust behavior (if needed).
Netezza (Data Warehouse) provides a location where "fast" analysis on a "limited" subset of the data can occur.
Hadoop holds everything else so that longer term analysis with full data sets is possible. This could be used to:
Adjust the control models
Change which data subsets are warehoused
Perform ad hoc deep dive analysis.
Perform regular analysis on data sets which are too large to reasonably warehouse (e.g. raw scan data).
Once you have a system which can process huge amounts of data (big data), you need a place to store all of that data. This is what databases are for.
Traditionally, this has meant a relational database. But relational databases place many constraints on how the data is modeled ("normal forms") which are inconsistent with the high volume data sources which need to be analyzed (e.g. all the webpages in the world, all the legal documents in your company or all the tweets being posted each day).
Relational DBs require that data be modeled into a set of tables that contain unique entities (rows), described by attributes (columns) which are arranged in such a way as to describe one aspect of each entity in each table with no redundancies.
said another way:
each row has a primary key made from one or more columns. Column data contains single values (1NF).
All columns in a table relate only to the complete primary key (2NF)
All columns in the table contain data which is not derived from other columns in that table (3NF)
To add more columns which do not fit these constraints, you must put them in another table and join them together.
said yet another way: The key, the whole key and nothing but the key.
These restrictions allow for optimal query structuring and performance while minimizing anomalies due to data changes. However they do not easily support the lack of simple structure between the contents of many data sets.
Non-Relational (NoSQL) DBs remove the restrictions on data normalization and focus, instead, on optimizing around data that does not fit well into the normalized structure which relational DBs (mostly) require. Because there are different analyses of interest and different data sources which "best" embody the data of interest, there are different types of NoSQL databases.
Below is a diagram showing the various database types.
Key Value (aka Big Table)
Data is stored in a GIANT ordered table of rows and columns.
Rows and columns still serve the same general purpose as in a relational DB case
rows = unique entities
columns = attributes.
...but normalization is not required (or expected)...
Data may be sparsely populated in the columns.
I.e. a given row may only have data values for a small fraction of the columns (because most the columns don't apply to the entity this row describes).
Columns may be VERY large in number and depend on what the DB is structured to query for.
e.g. all unique word pairs for the entities in the database
Google originally developed this technology for searching through web pages to fulfill search criteria. Roughly speaking:
rows = web pages
columns = search terms
Document
Entire documents are stored in a searchable format.
Queries search through the documents to identify the information of interest and return statistics or the document IDs.
Good for finding actual documents which contain specific information or summarizing the information contained in a set of documents.
Graph
Stores information about relationships between entities (objects) in the DB
Good for finding objects that are related to each other according to certain criteria.
e.g. find people (entities) who are members of the YMCA (another entity) who lived in New York in 1999.
How does this relate to Big Data?
Many NoSQL DBs are built to operate on distributed file systems and process queries via distributed computing. In fact, the very nature of the data being looked at is so large
NOTE: The MRD should describe WHAT needs to happen overall and between parts. The MRD should not (usually) describe HOW all the parts get implemented - that is for the engineering design document.
Describe the end-to-end scope of the problem to be solved
Break the problem into logical sub-problems
Describe the inputs required to resolve each sub-problem. This includes:
human interfaces for data input
one time
interactive / iterative
machine / data inputs from external data
machine / data inputs from internal (transient) data
Describe what output should be generated by resolving each sub-problem. This includes:
which data is needed as "the" output. i.e. the "permanent" data.
What is the expected input format of the consumer(s)?
which data is needed to address another sub-problem. i.e. transient data.
All of this should be written with an eye to the system in which the functionality described by the MRD lives.
Every input is the output of another system, ideally described by an MRD (reference it if you can).
Other systems may need the output of the system described by your MRD. Include these systems as examples in your MRD to give color to the bigger picture problem being solved.
Human input interfaces (User Interfaces), Machine Input interfaces (APIs) and Permanent Data stores (HDDs or Databases) may be shared between multiple systems. If they are, or should be, note that explicitly.
One obvious challenge, given the recursive approach to MRD writing given here, is figuring out where to stop.
How big should the scope of THE problem be?
My experience: when in doubt, make the scope too big. Then scale back the scope during reviews based on feedback from the stakeholders.
One of the ways that Big Data of the sort discussed in this last post is implemented is with an open source technology stack called Hadoop.
Hadoop consists of a two main parts:
HDFS - Hadoop File System
MapReduce infrastructure
These allow data processing jobs to be divided among multiple nodes and then aggregated into a single result. In essence, this constructs a large, parallel computer from many smaller computers - basically the opposite of virtualization.
Schematically, a Hadoop cluster looks like this:
10,000 ft view of how it works
The Job Tracker on the master server gets a job
The Job Tracker breaks up the job using the map function
Basic queuing ensures that any one node is not overloaded with tasks
The are tasks preferentially distributed to the nodes nearest the data on which the task must operate to minimize file transfer overhead*.
nearest = same node as the data resides.
next nearest = different node but behind the same switch (so that data transfer is localized to that network segment).
The Job Tracker gets status for all tasks via the Task Trackers as they run.
If a node stops reporting, the Job Tracker will redistribute that node's tasks to another node.
When all tasks are complete for a job, the Job Tracker has the nodes execute the reduce function to generate a single result from the tasks' output.
The final output may then be used by other applications directly or as the input to another MapReduce iteration.
I am certain that I missed almost every important detail in terms of the actual engineering implementation of a Hadoop cluster. But for those who just need a cursory understanding of the technology in order to make sufficient sense of what engineering is actually talking about to sanity check proposals, I hope this hits the mark.
--------------------------------------------------------------------------------------------------- * HDFS and the awareness of data's physical location is very important in dealing with large data. To make compute and storage capacity scale linearly with cost, the data must be spread around all the nodes in the cluster and a record kept of where all the data actually is. It can't be kept centrally or data transfer becomes a serious bottleneck to computational performance. I.e. it could take longer to move the data to a compute node over a network than it takes to actually process it.
This distribution is managed by HDFS.
HDFS holds and replicates data in the system in order to minimize the chance of a bottleneck. It does this by keeping at least 3 copies of the data:
an original
a copy on another node behind the same switch as the original
a copy on another node on a different switch from the original
This replication attempts to strike a balance between:
keeping enough copies of the data to minimize the queue size on each node (to ensure timely completion of jobs) and to ensure robust execution despite failed tasks on dead nodes
-and-
reducing the total storage capacity of the system by duplicating data.
The inspection tools that I work with are capable of churning out enormous amounts of data - on the order of terabytes an hour. To handle that data volume, we have done what every company did (up until now) which was sample from the data and reformat it to fit into a gigabyte sized database. This lets the data be accessed for useful analyses but creates a problem in that much of the data is actually lost, ultimately limiting what can be learned.
This is traditional data processing.
To store and handle more data, we swap out the existing hardware with bigger (read: more expensive) hardware. This works only up to a point as the cost of bigger h/w does not rise linearly with capacity. So you reach a limit to what is cost effective pretty quickly.
Schematically it looks something like this:
Several data sources structure the data and put it into a database. Programs running on the compute resources access the data from the database and provide some analysis. Scaling the system means getting bigger h/w.
Big Data changes how this can be done.
At its heart, Big Data is about making the data storage size and computing power scale in a linear way with cost. This is done using a few technologies which I will describe in more detail later.
Schematically it looks something like this:
One system coordinates the actions of many nodes in order to generate a desired computing result. Each node contains both compute and storage. The entire system works in the same basic way regardless of how many nodes are present. So if more data storage or more computing power is required it can be added by provisioning more nodes instead of replacing the entire system with larger nodes. This makes it easy for a company to scale its costs with actual business volume or to handle burst loads via a hybrid cloud approach (i.e. provisioning additional nodes on demand as an IaaS offering) to avoid large capital expenditures due to over provisioning.
Velocity : Analyzed in seconds rather than in minutes, hours or days.
Variety : Coming from many sources, including unstructured data sources (i.e. things that don't fit into a relational database very well).
So, instead of throwing away most of the inspector data as we do today, we could keep the data and build a system at reasonable cost which could actually process it. With hard work on new algorithms which could take advantage of the new data would come new insights into the phenomena behind the data.
I got to thinking about some of the problems I see at work around software development & roadmap and decided to apply some systems thinking to the situation. This is the result.
The key learning:
Really good use case validation is probably the largest leverage point.
Agile development can be an alternative to really good use case validation.
No surprises there but it is interesting to see the dynamics that lead to those conclusions. The feedback loops suggest alternative paths to address the customer acceptance problem when neither use case validation improvements nor agile development are feasible. For example:
What if you refused to add late features and managed the initial urgency to gain product acceptance? As long as the gaps are fixed in the medium term, the improved roadmap credibility may be enough to gain acceptance in the face of gaps next time because the customer believes your roadmap claims.
If apps and product managers are failing to validate use cases sufficiently, can you increase scrutiny on requirements by engineering and increase insistence on complete test case details by SQC to minimize factors which cause schedule slips and perhaps offset the slips caused by feature adds?
This is what happens when you dedicate a few hours to intensively trying to answer a single question about a single company (in a slightly modified version to remove work related info).
Thanks to Kwok Ng for his help on this.
By the way... Does anyone know a GOOD way to get PPT slides onto the web with no conversion artifacts?
Neither SlideShare nor Google Docs did it for me.
I have been learning a bit about building management systems (BMS) and realized that, abstracted the right way, the manufacturing execution systems (MES) of semiconductor fabs are pretty similar in overall concept and components. That parallel made it much easier for me to frame my learning. 
.png)
