A new direction
Embracing the user-centric approach, Morpient delivers an interactive big data analysis capability in our SaaS product that empowers users with the following:
Genuine self-service
Gleaning insights independently, at speed-of-thought.
Petabyte-scale interactivity
Data-centric decision-making and analysis at scale.
Lowest TCO
Budget competitive, including open source platform adopters.
We describe the underpinning technogy, for the above pioneering achievements, as follows:
Describe “Intelligent decision making” — a data-driven decision making. Exemplify it using a conversational (e.g., ChatGPT) search based decision making.
Identify the guiding principles for intelligent decision making.
Use these guiding principles to describe the four pillars of the technology.
The process of data driven decision making is termed as ‘Intelligent decision making’ where user and cloud work synergistically to significantly enhance decision making. This is done by:
Intelligent Decision Making
A conversation-search based example of intelligent decision making:
A businessman visiting Delhi had to delay his stay by few days. Using conversational search (ChatGPT, Gemini etc.), he was looking to see how to spend the extra time. Suggestions were plentiful and after few refined subsequent questions, he narrowed his search to Mughal related sites.
Besides giving the user options within Delhi, conversational search also suggested a trip to Agra (1 hour flight) to see the Taj Mahal, which he had never imagined. Through conversational search, the decision to see Taj Mahal was chosen as the optimal choice.
The ability for the user to ask questions, review the response and follow up immediately with more refined questions is crucial to intelligent decision making.
This decision process was proposed decades ago as ‘Intelligence Amplification’ by D. Engelbart, W.Ashby, and others.
1. Pose an initial query
Searching for initial information to make a decision.
2. Peruse the answer
To see if the information can support making the decision.
3. Modify the query
Iterate until enough information is found to make an informed decision.
Intelligent Decision Making
Guiding Principles
Interactively get answer to peruse
Get the answer at speed-of-thought.
User makes the decision
User leverages his/her enterprise-tribal knowledge to decide if information is enough.
Interactively modify query
Repose the query to follow the train of thoughts.
Obviously, traditional big data analysis, having to rely on a data team, CANNOT enable interlligent decision making.
Clearly decision making with conversational search adhere to all the above three principles.
More importantly, if conversational search based decision making were required to use a data team to pose question, the decision making process will fail miserably — even with all the advantages of the power of LLM.
Thus we conclude that the above guiding principles are very important for intelligent decision making and use them to describe the technology.
The Morphient SaaS Offering
Intelligent decision making the right way
The intelligent decision making, using big data, is made possible by the following four pillars of technology.
1. Story-as-a-document: Visualized view of data + Interactively editable
2. Interactive Analysis: Unlimited data. In the browser.
3. Genuine self-service: User interacting with data directly.
4. Massively parallel execution: Unlimited computing power, just when needed.
These are described below.
Visualized view of data + Interactively editable
Story-as-a-document
The visualized data is edited like a document, where a sequence of edits show changes immediately. This interactively-editable, visualized-view of data is called Story-as-a-document. Story refers to all possible analysis that an be done using that view.
Simple Interactivity
Allows the user to browse the story from any angle to analyze the data as well as editing the query or visualization to get a refined story.
Easy to Use
Leveraging user awareness of common functions like ‘edit’, ‘undo/redo’, ‘copy/move’, ‘delete/change’, and ‘filter’ etc. allow quick user onboarding.
This model enables the business user to query & analyze and then modify their queries or visualization by themselves. This provides the immediacy factor crucial for intelligent decision making.
Interactive analysis
Unlimited data. In the browser.
The story-as-a-document model serves as a vehicle for users to be able to interact with unlimited data by leveraging the in-browser query engine.
JITQBE
To query unlimited data, we devised a proprietary data structure called JITQBE.
This is a curated subset of data optimized for that specific story based on user requests.
JITQBE stands for Just-In-Time QBE (pronounced as cube) behaves like a data-cube but without the exponential size problem
Query Engine
Query engine leverages the JITQBE in the browser memory to deliver an interactive analysis experience.
This JITQBE assures an interactive experience without time consuming round trips to the cloud for each interaction.
This level of interactivity, assured by the optimized JITQBE, enables users to analyze at speed-of-thought.
Recomputation of JITQBE
As user changes a story fundamentally, it automatically triggers re-computation of the JITQBE in the cloud.
A new JITQBE for the modified story is delivered, in matter of minutes, so user can continue the interactive experience.
User interacting with data directly.
Genuine self-service
The elimination of intermediaries to operate the system allows Morphient SaaS to be leveraged by users directly, without the delay due to the external data team.
Analytics
The user can interactively slice & dice the data themselves, and see the data visualized. Both the query for visualized data and the visualization can be edited.
Automatic Optimizations
Just-in-time re-computation of JITQBE eliminates the need for someone else to optimize the execution, as the data scales.
No Maintenance
All cloud operations run on lambda with zero compute instances provisioned. Therefore lambdas are the only resources to be managed. These are monitored in the cloud by Morphient.
Any reliance on an external team to curate the data defeats the intent of self service. We refer to our offering as ‘genuine self service’ to emphasize a true no-intermediary requirement.
Massively parallel execution
Unlimited compute power, just when you need it.
Processing unlimited data requires unlimited compute power. More importantly, when the user is waiting for a reply, massively parallel computing results in a high degree of responsivenes.
Interactive experience
Assuring interactive experience being a lynchpin goal for intelligent decision making, we have leveraged massively parallel execution to achieve this goal. This includes instances like re-computation of JITQBE.
Cost effective computation
Massive computing power needs to be leveraged judiciously in order to assure our lowest TCO goal.
Some techniques leveraged include computing JITQBEs from other JITQBEs and Just-In-Time provisioned lambda services for all computations.
All our computing in the cloud are achieved using unlimited parallel lambda execution. These lambdas are procured (and dismissed) just in time, which avoids the idling of expensive computing resources, to achieve our lowest TCO objective.