The practice of data science and analytics has undergone a sea change over the last ten years. Here’s a 100,000 feet view of the changes that we have seen across various stages of the data – insights lifecycle, and how they are improving organizational abilities:
1. Data Processing: The evolution of Hadoop, columnar storage, No SQL and the related animal farm of technologies has made it possible to capture, store and analyze every byte of data and is already leading to improved efficiency and productivity in business processes
2. Algorithms: Huge progress have been made in artificial intelligence that has led to new solutions for traditionally difficult problems, such as image recognition or fraud prediction
3. Cloud: Cloud computing has removed infrastructural barriers for everyone, from large enterprise to individual innovators, to harness the power of cluster computing to create solutions to difficult computing problems
4. Computing: Advances in GPU computing and upcoming advances in quantum computing is enabling us to solve problems that either take too much time or are simply intractable with today’s computers
5. User Experience: Innovative visualization and browser-based delivery have made it possible to simplify the delivery of insights, making it more actionable, thereby amplifying individual and organizational capabilities
6. Explosion of Data: Finally, we are on the cusp of an IoT era where embedded sensors connect every device to the corporate computing network, blurring the boundaries between the digital and the physical, with the ability to make optimal, data-driven decisions at real time
These changes are not happening in isolation, but they are feeding off each other. For example, improved computing advances is leading to the deployment of new class of algorithms for solving problems that were very difficult until recently.
The pace of change is accelerating across all of these areas. New data processing methods and technologies are emerging across all these areas. For instance, we’re seeing the adoption of Niagra Files (NiFi), which is a secure way of managing complex data flows across the enterprise. Several other data processing technologies are being created every day. New algorithms are being created, in an open and collaborative fashion, that advance the state of the art in artificial intelligence. Cloud vendors are adding greater capabilities, quantum computing is becoming a reality, there are significant improvements in user experience with the rise of JavaScript frameworks, D3, mobile computing, and IoT adoption is accelerating.
At LatentView, we are grappling with all these changes everyday, while solving day to day analytical problems for our clients and helping them transform their business processes. It’s impossible for our client delivery teams to stay on top of these changes, while serving our clients. Hence, we created the IdeaLabs team, which is a dedicated team focused on innovating around emerging technology areas.
The IdeaLabs team’s mission is to track and understand the major technological changes, the problems that they solve and the benefits that they bring, and harness these technologies to create solutions and accelerators that add value to our clients. Through this blog, we hope to connect with the rest of the analytics community and our clients.