As Businesses realize the imperative to capture not just structured data but also unstructured data to sustain their competitive edge, they will see themselves looking more towards adopting Big Data tools.

Big Data continues to surge across sectors, making it imperative to deploy tools that can manage and analyse the unprecedented volume, velocity, and variety of the data available.  The Big Data market reached $18.6 billion in 2013, a 58 percent growth rate over 2012, according to Wikibon Principal Research Contributor and Big Data Analyst. Further, Big Data-related services revenue makes up 40% of the total, with hardware at 38% and software at 22%.

While relation databases and traditional BI tools have been equipped to deal with transactional data generated and help companies in their Business decision making, beyond that unstructured data was largely not stored but which is now feasible to capture and therefore becomes a further source of information that can be utilized. Weblogs, social media, email, sensors, and photographs are examples of less structured data that can be now mined for information.

“Decreasing cost of both storage and compute power have made it feasible to collect this data – which would have been thrown away a few years ago. As a result, more and more companies are looking to include non-traditional yet potentially valuable data with their traditional enterprise data in their business intelligence deployments,” says Aydin Gencler, Director, Cloud OS Data Platform at Microsoft Middle East and Africa.

He adds, “Today, enterprises want to capture and manage large amounts of information from multiple sources both structured such as ERP, CRM and database systems and unstructured including web sites, social media and streaming. By investing in Big Data solutions, enterprises would be able to understand the semi structured or unstructured data that represent more than 85% of their business will gain a huge competitive advantage over their competitors.”

There is a steadily growing confidence in Big Data products and services in the enterprise segment. Vendors are striving to get the message out into the channel and customer base to increase the adoption rates.

“The surge in data is a global phenomenon. Even in the Middle East, we continue to witness an accelerated data generation as a result of a surge in Internet usage, online banking and social networking in addition to smartphone and tablet penetration and this is only going to continue. Increasing sources and amounts of data are being created, offering organizations more insight into their decisions, ideas and predictions,” says Sadi Awienat Chief Technology Officer and Global Services Lead, Gulf and Pakistan, EMC.

Therefore the traction in the region needs to be a lot better though it is restricted to a few Businesses at the moment although the traditional BI and analytics deployments have been widespread.

He adds, “In a maturing market like the Middle big data presents huge opportunities for enterprises to better understand their customers, evaluate larger market trends and predict emerging business opportunities across the region. Business Intelligence and Analytics have always been key priorities with CIOs in the region and few businesses have already begun harnessing Big Data Analytics, with more and more to follow path.”

One of the bottlenecks could be the readiness of customers in the region to consider Big Data deployments as they may already have BI tools in place that are doing reasonable jobs of mining and analyzing structured data.  These customers may hold back until they are certain they really need the information from the additional unstructured data but in certain key verticals, customers are realizing the value they can extract out of such Big Data deployments.

Boby Joseph, Data Practice Head at Data Science Technologies, a subsidiary of StorIT Distribution and a pioneer in the Big Data Analytics solutions and services space says, “Though organizations in the Middle East are becoming aware of the challenges Big Data brings, the adoption of Big Data analytics in this region is still lagging far behind as compared to the global adoption rate. The region definitely needs to deploy Big Data tools for enhancing the analytics and reaping the benefits of data mining. However, companies in the Middle East are beginning to realize the benefits of harnessing big data in business, education, government and security.”

Driven by customer and market demands moving at breakneck speed, businesses need to engage with, capture, manage and transpose data into intelligent action to stay relevant. There is a key role for the VAR channel to engage with the wider customer base and understand their apprehensions if any about Big Data and provide consultation that can convince them to adopt Big Data tools.

Andrew Calthorpe, CEO at Condo Protego, a leading Systems Integrator in the region says, “MENA’s VARs would do well to connect with their clients and start to answer some of these questions with a positive outlook.  Top priority is to instill the mindset that data should never be dismissed as forbidding white noise, but rather treated as one of the most valuable assets in the game.  .”

He adds, “A meticulously planned, cutting-edge and scalable IT architecture that can collate and store enormous amounts of unstructured data is now a must.  A whole lot of consultation and integration needs to take place across the region, and VARs need to be ready, willing and able to help make it happen.”

Challenges exists however for companies looking to invest in Big Data solutions. According to the Wikibon Big Data analyst, barriers exist which include a lack of best practices for integrating Big Data analytics into existing business processes, concerns over security and data privacy, continued “Big Data washing” by legacy IT vendors, a volatile and fast developing market, and a lack of polished Big Data applications for solving specific business problems.

The Hadoop open source storage system that helps collect large amounts of data from servers and breaks into manageable chunks is seen as a key technology underpinning Big Data tools from major vendors. Because it is schema-less, Hadoop can ingest structured or unstructured data from any number of sources. This data can be collated in any combination to enable incisive decision making. Further, the introduction of YARN (Yet Another Resource Negotiator), a resource management layer to Hadoop in 2013 is seen as a milestone as it provides the structural foundation for Big Data analytics to move beyond MapReduce-style batch processing.

“Microsoft embraced Hadoop project for many years and started building its integration on the Hadoop platform to provide a Big Data solution that is available with SQL Server, SQL Server PDW appliance and Windows Azure platform as Azure HDInsight. Microsoft’s approach is to standardize implementing Hadoop on Windows Server on premise or Windows Azure in the cloud while simplifying usage and extraction of data via HDInsight and technologies like Polybase in SQL Server PDW. So far Apache Hadoop has proved to be the best option to implement Big Data,” says Aydin.

He adds, “Microsoft believes big data should be in the hands of people who are closest to their business. The approach is simple—combine the power of 100% Apache Hadoop with the core databases and bring unstructured and structured data to life through rich 3D data visualizations with the tools that your business uses most.”

Microsoft’s Big Data solution that offers a modern data management layer that supports all data types – structured, semi-structured and unstructured data at rest or in motion.  There is also an enrichment layer that enhances your data through discovery, combining with the world’s data and by refining with advanced analytics. Finally, there is an Insights layer that provides insights to all users through familiar tools like Office. Microsoft has built rich, 3D data visualizations and storytelling right into Excel. This makes it easy to slice, dice and visualize multiple data sources – even modify them on the fly while presenting in PowerPoint – helping you uncover insights that would have otherwise remained hidden in static charts.

HDInsight is Microsoft’s new Hadoop-based service, built on the Hortonworks Data Platform (HDP) that offers 100% compatibility with Apache Hadoop. HDInsight enables customers to gain business insights from structured and unstructured data of virtually any size and activate new types of data irrespective of its location. Rich insights from Hadoop can be combined seamlessly with the Microsoft Business Intelligence (BI) platform to give customers the ability to enrich their models with publicly available data and services using familiar tools like Office and SharePoint.

There are several variants of Hadoop based solutions that the channel can work with to offer solutions to customers. These include Cloudera, MapR and Hortonworks. IBM as well Pivotal, an EMC spin-off also offer their Hadoop versions. Many of the data management vendors would be working with one of these Hadoop distributions or with several of them.

DST focuses on offering consulting and integration services in the Big Data Analytics and is allied with vendors including SAS, Hadoop, MapR, MapReduce, Greenplum, Elastic Search, Python, MongoDB, Redis Solr etc..

Boby says, “Data Science works on most of the Big Data platforms in Open Source. All Solutions and architecture designed can incorporate various variants of Hadoop. We also have integrated Cloudera Hadoop offering where in the customer has a choice to opt for having it integrated (factory installed) or to choose from choice of other Hadoop variants. Data Science has the most exclusive Hadoop offering from one source. Data Science can also deploy Hadoop in the Microsoft Windows environment for various Microsoft Analytics deployments. Microsoft has some very powerful analytics in the Big Data space and Data Science has the skill set to deploy in similar environments.

According to SAP, Big Data’s vast potential is one of the main driving forces behind the growing popularity of SAP HANA, an in-memory platform that combines transactional and analytical processing in one system. SAP HANA provides a single platform for advanced real-time analytics, data warehousing, visualization and reporting and enterprise applications. The platform offers a choice of deployment models and partners, providing lower cost and faster innovation from an open ecosystem.

Paul Devlin, Director – SAP Platform Solutions, SAP MENA says, “Big data is characterized by the three V’s: velocity, volume and variety. Businesses must think about architectures that can accommodate big data from access, storage and consumption perspectives.  Firstly, businesses must be able to access any data, in any format, at any speed.  Secondly, businesses must be able to effectively and economically store and access big data.  Here the key is to deploy appropriate platforms based on the temperature (access frequency) and value of data.  Finally, businesses must provide easy and intuitive ways for analysts and decision-makers to consume big data in order to make sense of it and make better, faster decisions.”

He adds that depending on overall volumes being accessed, businesses should think about hybrid approaches that utilize in-memory databases such as SAP HANA for hot, high-value operational data; high performance disk-based data warehouses such as SAP Sybase IQ for warm and historical information, and possibly also Hadoop clusters if the volumes of data make it uneconomical to store data in-memory or in disk-based RDBMS’s.   SAP has made it possible to integrate HANA and Sybase IQ with Hadoop for easy management and access of data across the different storage tiers.

SK Solutions, anti-collision software pioneers, has partnered for instance with SAP to enhance worker safety, reduce costs and improve productivity on Dubai construction sites. The project uses sensor-based data fed through a system using a portfolio of SAP technologies, including in-memory computing platform SAP HANA, to prevent cranes and construction vehicles from colliding.

Dr. Séverin Kezeu, CEO, SK Solutions says, “Our solution entails deploying sensors on cranes and construction vehicles to pull actionable data such as 3-D motion control via inertial motion unit, location via GPS and load weight, equipment usage and wind speed and direction. This information is then extracted to help keep personnel safe and enhance utilization of construction equipment, which helps improve productivity and ensure projects meet key milestones.”

SAP HANA also allows users to easily analyse billions of records in seconds, which means that complex business questions and simulations that took hours can now take mere seconds.

Cloud based offerings

Cloud based solutions could attract mid-market customers who will look at lower cost of ownership. Cloud based offerings are available and are expected to become more pervasive in the Big Data analytics segment.

Microsoft’s Aydin says, “As a result of exponential data growth compounded with mobile access to data and applications from multiple devices such as smart phones and tablets and increasing social media adoption, we expect the cloud computing will grow to address the needs of enterprises and consumers. Therefore, we expect cloud based implementations will grow to handle the data growth, storage and access to Big Data.

He adds, “Cloud based solutions from simply storing data in the cloud to gaining insight will be consumed as a service. Therefore, the total cost of software and hardware ownership will decrease and become a subscription based service model that would be much more affordable for SMB.”

Through cloud based subscription models, Big Data can gain some needed traction with SMB customers. SMBs will be able to quickly adapt to the cloud services to conduct and move their businesses forward with a low Capex and OPEX.

Boby says, “Cloud would play an important role with the SMB segment that can save on investment to slowly move their needs of analytics and talent to different geographies and thus pay-per-use model integrated into their current needs. The elasticity of the cloud makes it ideal for big data analytics — this practice of rapidly crunching large volumes of unstructured data to identify patterns and improve business strategies will help big data adoption of cloud. Data Science Technologies has various Cloud-based Big Data solutions available for the SMB segment, which we can share with companies that are interested.”

The road ahead

More and more BI tools will incorporate Big Data elements in future. As the traditional BI tools deal with the structured data sources, these tools will either incorporate the new enhancements to understand Big Data or complement with the new tools that can already work Big Data.

Aydin adds, “The traditional BI tools and Big Data tools will be complementing each other to emerge as the next generation business analytics. The solutions that provide a hybrid solution that embraces both on premise, traditional, structured as well as in the cloud, semi or unstructured data access and analysis will have a better chance to address customers’ needs.”

While BI tools are quite competent to deal with the structured data that Businesses have been used to, the fact that Big Data tools that help mine through the data deluge that is coming in will make it imperative for most BI vendors to include Big Data variants, especially as customers start demanding such solutions. That day may not be far off.

Boby Joseph opines, “Big Data will replace some of traditional BI tools but, BI is still part of Big Data. However, new data sets have taken them to a new dimension, where in existing BI tools or applications are deficient in servicing the consumer’s voracious data mining and visualization needs.”

In summary, Big Data is a segment that has taken quite some time in gestation when it comes to having  and we are yet only at the beginning of what certainly appears to be a strong growth area. 2014 will see more products coming in and increased adoption which however will still be only a modest growth considering the fact that Big Data analytics is expected to be quite pervasive eventually, especially when Technologies like IoT (Internet of Things) will make more things around us intelligent with embedded sensors that can transmit actionable data back.