Data Integration

Internet of Things

How the Internet of Things is Reinventing the Energy Industry

How the Internet of Things is Reinventing the Energy Industry 650 486 Exist Software Labs

Internet of Things (IoT) has given so much convenience to human activities and transactions over the years that it has become one of the most important technologies on a global scale. It helped not just individuals on its day-to-day activities; but also various business verticals on its company-wide operations.

The energy industry in the Philippines is no exception. The Internet of Things (IoT) provides a wide range of applications that can be utilized in the energy sector; this includes energy production, transmission, distribution, consumption, and many more.

See figure 1.1 (This is the evolution of the Internet of Things (IoT) over the years.)Power System Solutions

In previous years, most of us, consumers, never really considered mother earth when consuming energy; This is one of the improvements that IoT may bring forth! Organizations can drastically cut waste thanks to increased control over energy consumption. It brings significant cost savings and allows for a reduction in CO2 emissions, which benefits the environment.

Another challenge that the Energy Sector has is the lack of integrated energy planning and demand forecasting. This significantly widens the gap between energy supply and demand; However, with the help of IoT’s advanced analytical systems, organizations can now develop precise and real-time forecasts. Here are some Internet of Things (IoT) capabilities that will reinvent the Philippine energy industry.

See figure 1.2 (Here is the entire IoT ecosystem of the Energy Sector. From generators, transmission, distributors to the consumers.)

Power System Solutions1. Intelligent Grid

IoT enables grid systems to gain control over power flow or significantly curb energy use and helps in tracking real-time energy consumption that gives users almost complete control over their energy consumption. Its automated approach also helps consumers become more cost-efficient while decreasing energy waste.

While Energy companies, on the other hand, may achieve real-time monitoring, rapid data restoration, and highly accurate usage forecasts, making energy system management and maintenance much easier.

Empower your system with IoT today!

2. Monitoring of Processes and Resource Optimization

Automation is one of the benefits that IoT can provide; having an IoT system means less human intervention, and less human intervention means fewer errors. The use of sensor devices in a power plant allows for the automated execution of processes and the delivery of better, error-free production. The availability of data on all areas of the supply chain enables the operating system to make accurate judgments on a wide range of factors.

IoT solutions generate real-time data, allowing management to act quickly and take appropriate action if there are errors or inconsistencies, allowing businesses to produce higher-quality services.

3. Analytical Advances

With increased energy consumption demand, energy businesses must rely on advanced analytics more than ever to meet customers’ expectations for more options, better transparency, and lower energy prices and waste. In order to compete, utilities and power generators must utilize the same analytics to precisely anticipate future power usage.
The sensor-enabled technology assists the Philippines’ energy sector in gathering more precise data that can be used in proper forecasting of energy supply and demand. As a way, it lowers unnecessary energy usage.

4. Reduction of maintenance and operating costs

Predictive maintenance is another significant advantage of IoT data. Proactive testing and repair significantly reduce the amount of time the machines are inactive, hence minimizing catastrophic equipment failures and lowering maintenance expenses. It enables managers to take complete control of energy data from the beginning and considerably optimize the operation. The use of an IoT-powered solution in the energy sector employs sensor-based approaches to automate the industry’s operation.

Intelligent sensors and devices, for example, can communicate information from distant equipment to detect an impending fault, minimizing damages and stoppages and giving the system much more flexibility.

5. Increase the Production of Renewable Energy

A smart, dependable, and efficient power supply contributes to the smooth operation of smart cities. The application of IoT in renewable energy generation will assist them in producing enough energy to meet their needs.

Sensors attached to generation, transmission, and distribution equipment are used in IoT applications in renewable energy production. These technologies let businesses remotely monitor and control the operation of their equipment in real-time; This reduces operational expenses and energy waste, but more importantly, it helps us minimize our reliance on fossil fuels.

Exist Software Labs will continue to assist businesses in the energy sector

You may now achieve the above-mentioned processes and technologies with Exist Software labs! For over 20 years, we helped several energy enterprises by implementing multiple Energy Automation solutions.

Achieve advanced analytics without breaking the bank, optimize your operations, and achieve advanced management systems by utilizing various enterprise-grade technologies designed exclusively for the energy industry.

Empower your system today!

Learn how to fully automate your processes to create a more competitive, transparent, and efficient system with our Power System Solutions.
Take your power system to the next level!

Energy Tech Trends

2022 Energy Tech Trends to watch

2022 Energy Tech Trends to watch 800 507 Exist Software Labs

The COVID-19 pandemic, numerous heavy typhoons, and other unfortunate events affect all business sectors in the country. One of the major industries affected is Energy and Utilities, and it highlighted the necessity for The Energy Industry to adopt a sustainable perspective and improve the technology system.

With the rising demand for energy in 2022, the Department of Energy will continue its advocacy to produce renewable energy to cut market prices and achieve sustainability. On the other hand, the private sector will continue to develop technology systems to achieve efficiency, high effectiveness and keep up with the continuously evolving energy sector.

While numerous technology solutions can assist these firms in achieving their digital objectives, only a few are expected to have a significant influence in 2022. These are the 2022 Energy Tech Trends to watch!

1. Powering Digital Economy Through IoT (Internet of Things)

IoT, or the Internet of Things, has played a vital role in advancing digitization in several industries, including IT, energy, agriculture, healthcare, and many more.

It is one of the most advanced technologies, and one of its advantages is that it improves the efficiency of several businesses, including energy. And as for the energy sector, one of the most important functions of IoT is energy conservation. 

The Internet of Things enables electricity firms to read data in real-time. It enables them to quickly gather, calculate, and analyze data to improve decision-making. It also helps the energy industry transform into an integrated system, resulting in a smart solution that is equipped with advanced technology to increase industry value and maintain asset efficiency for the benefit of the economy.

2. Fifth-Generation Technology will establish the connection

Many companies will continue to improve their systems in 2022. The sector will continue to advance the electric grid to make it more reliable and less expensive, thanks to the national government’s directive to push for renewable energy and industry’s developing market demands.  

These companies rely on Millions of connected devices and digital systems, such as smart meters, sensors, management systems, to communicate data from many locations. And with their objective to digitize their system, they need to have fast and dependable technology, thus creating a need to access 5G technology. 

5G technology is the next generation of cellular technology after 4G. It has faster speed, lower latency, and the capacity to connect more devices at the same time.

Fifth-generation wireless technology will provide new features and more efficient smart grids. New 5G mobile networks will assist the integration of unconnected devices into new smart grids; it will also help the development of new electricity load forecasting software for accurate energy monitoring and forecasting. Organizations will now be able to receive and process the massive volume of data at quicker speeds with no chance of downtime.

3. Companies in the energy industry will continue to migrate to the cloud

The cloud holds the potential for endless growth, system efficiencies, and digital integration in any business industry.

With the Power industry’s continued growth, it needs a system that can handle its complex process and massive data efficiently, effectively, and precisely; this is where the cloud can help.

The cloud has the potential to change every aspect of the energy value chain. Connectivity, scalability, analytics, and automation can all help you save money and increase profits in countless ways.

Thanks to Exist’s strong foundation in the power industry, with its business solutions to industry market leaders and cloud services to other business verticals, we can now quickly apply tailored advancements to your company.

4. Artificial Intelligence will revolutionize the game.

Artificial Intelligence (AI) is becoming relevant in the energy industry and has great potential for future energy system structures.

Digital technologies such as Artificial Intelligence (AI) will make energy sector systems more intelligent, efficient, dependable, and sustainable, which benefits the entire energy sector chain, from generation to transmission to distribution to the consumer.

In terms of alignment with the government’s ambitions, AI would also benefit Renewable Energy. With the growing use of renewable energy sources, it is becoming increasingly difficult to regulate the megawatts that are fed into the grid; with this, power networks will be unstable and prone to blackouts.

With this technology, renewable energy sources may now provide real-time, accurate data that allows AI to predict capacity levels.

5. Energy Sector will embrace the power of Machine Learning

With Machine Learning (ML), it’s as if you have a sophisticated human mind monitoring your system, complete with advanced self-learning algorithms, taking your data to a whole new level by making human-like decisions based on current AI data.

ML employs approaches that can be applied to predictive maintenance. Power lines, machines, and stations, in essence, are outfitted with sensors that capture operating time series data.

With enough data, your system can now forecast if a failure will occur in your system, allowing you to more efficiently monitor maintenance, reduce downtime, and avert system failure as soon as possible; thus, lowering your system expenditures.

6. Taking Advantage of Big Data

It’s one thing to get your data, but it’s quite another to use it to your advantage. 

Big Data Analytics has the potential to be a key driver in achieving optimal company performance in the energy sector.

Big data can help the energy business in many ways, including improved supply chain management, enhanced customer satisfaction, optimizing business efficiency, analyzing future risks and possibilities, and much more.

As a result, more and more energy companies are becoming more competitive. A superior business strategy that incorporates a large amount of data and efficient processes is assisting them in developing company value and increasing customer satisfaction.

So, how can you achieve the advanced system and follow Energy Tech Trends?

Whether you like it or not, the energy sector will continue to advance its technological innovation.

In this regard, it is important to look for an innovative partner who can add value to your organization, and this is where Exist Software Labs can assist you! With our extensive experience in the energy industry, we could bring you the innovation you deserve.

Empower your system today!

Learn how to fully automate your processes to create a more competitive, transparent, and efficient system.
Take your power system to the next level!

The 7 Wonders of Big Data and Analytics in the Power and Energy Sector

The 7 Wonders of Big Data and Analytics in the Power and Energy Sector 768 487 Exist Software Labs

Big data and analytics help the power and energy companies address and overcome new industry challenges through insights-based informed decision-making.

Nothing is forever. Like oil and gas, our natural energy sources are on the brink of scarcity, but global energy consumption is growing each fleeting day. The only way to earn extra time is to optimize its use correctly or gradually shift to renewable energy. To do that, big data and analytics help in automating and digitizing production, demand, and supply.

What is big data and analytics, you ask? IBM defines big data as any data that cannot be captured, managed, and/or processed by traditional management components and techniques. While analytics uses advanced mathematical techniques on diverse big data sets from different sources and in various sizes. In other words, data becomes big data when traditional methods, or a single computer, cannot be used to deal with data. It becomes analytics when these big data sets are processed.

The power and energy sector produces and collects large data sets continuously over some time. This data comes from their technology infrastructure, including sensors, cloud computing, wireless, and network communication. Smart meters alone in about a million households can collect 200 terabytes per minute.

Competitive edge of Big Data and Analytics

Businesses across the power and energy industry face many challenges to manage this data and draw insights from it. But when they start to invest in big data and analytics, data-driven companies will cultivate benefits in the long run. So, let’s look at the seven biggest advantages of implementing a big data strategy for power and energy companies.

1. Energy Efficiency

Energy preservation has always been a global issue. While there are advancements in using renewable and reusable energy, necessity still lies in saving it. But in today’s scene, companies should look more into using energy efficiently, requiring less energy for the same function, which is possible with technology.

By 2035, the Asia-Pacific Economic Cooperation (APEC), in which the Philippines actively participate, targets a reduction of aggregate energy intensity by 45%, based on 2005.

Big data analytics can play a significant part in the power sector’s role in this situation by deriving data coming from smart meters, asset operations, business policies, and weather data. It can be integrated and analyzed over time, which helps design electrical devices with energy-efficiency parameters, thus reducing power requirements. Energy efficiency plays a vital role in reducing carbon emissions. To achieve this, power corporations can forecast usage and predict savings by leveraging big data solutions in this manner.

2. Risk Management

Implementing a big data strategy in any business model can work wonders in assessing and managing hefty risks. It eliminates the practice of deriving business decisions from gut-feel and empowers executives with hard evidence instead. The truth can be said for the power and utility industry. Making informed investment decisions is the pinnacle of effectively utilizing the power of big data analytics.

Energy companies can derive valuable insights from power generation and consumer consumption. Predicting anticipated prices, creating dynamic pricing based on peak usage and demand, and adjusting the business operation model to cope with challenges can be done with data-driven decision-making. Enterprises that can embrace big data solutions’ advantages will have a strong market position and attain sustainable business growth.

3. Power Planning

Coming up with optimized power generation planning is another significant benefit of a big data analytics strategy. The strenuous task of matching power supply with the demand for energy on the grid, or economic load dispatch, can be performed quickly and efficiently by applying advanced big data analytics techniques. In effect, it improves energy production efficiency and lowers production costs.

4. Data Collaboration

Understanding what problems to address is first and foremost before implementing an analytics strategy. Big data analytics can predict production demand in the power sector, enhance efficiency, and optimize the operation process. Whatever it may be, enterprises can work with a wide range of data. Bringing all this data together can take a lot of time, but using big data analytics tools to collaborate all data can streamline the process and present visualized reports. 

5. Supply Chain Visibility

The retail industry utilizes big data effectively, especially in managing its inventory and forecast demand to improve supply chain management. The power sector is no different from retailers with its utilities. Technologies today, such as smart meters and smart grids, integrate with big data systems. This strategy enables enterprises to balance demand and supply, facilitate dispatch decisions, increase efficiency, organize inventory, and enhance reliability.

6. Customer Engagement

It is of vital importance for energy companies to be a data-driven and customer-centric business. Technology advancements, government regulations, and shifting customer behavior fast-track an optimized plan for consumer engagement. With big data analytics implementation, energy companies can shift to engage with customers in highly personalized environments, which can increase customer satisfaction, promote new products and lower the service cost.

7. Predictive Maintenance

Logical algorithms can consolidate and assess a large volume of data for power companies, enabling them to develop more effective business strategies. Energy providers can respond quickly in near real-time by predicting the market using specified vital indicators. Implementing big data will improve equipment monitoring and maintenance that will minimize production hours and system disruptions.

Transformation through Big Data and Analytics

With innovations such as predictive analysis, the power and energy industry is experiencing a large-scale transformation. With the assistance of predictive analysis, grids and meters are becoming more intelligent.

It is more evident today than before that this sector has a vital role to play in delivering essential products and services necessary to the world’s economy. Regardless of the industry’s shifts, power providers should continue to satisfy essential standards to provide energy services that are efficient, accessible, and sustainable.

Businesses continue to grow each day, but there are so many out there who do not have this technology and are not experiencing its benefits. There are real-time predictive analytics, grid operations analytics, and consumer analytics that power and energy providers can invest in.

Investment is mission-critical in this scenario to reap the real benefits of big data and analytics. Each day offers new opportunities, and the best solutions to today’s problems are yet to be done. The new normal calls for new innovations, and it is prime time for the power and energy sector to embrace this idea fully.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

Asia-Pacific: Fastest Growing Market in Big Data Analytics

Asia-Pacific: Fastest Growing Market in Big Data Analytics 768 487 Exist Software Labs

At a time when the pandemic forced enterprises to cut-down ICT investments, there remains optimism in anchoring big data analytics to improve business performance and derive highly informed business decisions.

About 3 in 4 enterprises in Asia-Pacific choose to keep investments in big data analytics (BDA) solutions or even expand this year, according to the latest COVID-19 survey by IDC. This standpoint is sound for businesses as big data analytics presents proven and essential use in enabling digital trust and resiliency now and post-pandemic, more so.

The coronavirus pandemic has caused a decline in the number of ICT investments. But businesses see technology resiliency as the best way to oppose its negative impact. By gathering relevant insights, enhancing productivity, satisfying customers, recognizing fraud, and reducing expenses, enterprises remain competitive with a data strategy in place. Moreover, enterprises are pushing towards public cloud deployments to optimize BDA investments.

The positive outlook around big data analytics comes at a time when global crisis forced companies to cut ICT expenditures. This decision became almost instinctive because of big data analytics’ proven value in keeping businesses alive amid the pandemic.

Energy sector’s fastest-growing market

In a report published by Mordor Intelligence, the Asia-Pacific (APAC) shows a promising future leading the energy sector’s market growth in big data analytics. The market intelligence and advisory firm cited the increased adoption and demand for IoT and smart technologies in government initiatives across the region.

If there is one thing that the energy sector can reap benefits from big data analytics, it is the power to manage scarce resources. After all, only 5% of the world’s energy consumption comes from renewable sources.

The most remarkable technology that is most likely to boost the market growth in 2021-2026 are smart meter systems. Smart metering in big data analytics involves components, such as grid operations, field services, resource planning, customer experience, and regulatory compliances. Big data analytics in smart metering helps in predicting energy consumption. Clearly, this empowers enterprises in managing supply and demand, and mitigating energy waste.

The same report mentioned India’s growing deployment of almost 2 million smart meters. Which takes advantage of today’s technologies and big data techniques. At the same time, China will be the leading country. Due to its increasing adoption rate of smart meters and smart grids across the country with its large population.

Banks and Telcos leading the pack

Contributing to nearly one-third of spending on big data analytics in the APAC region for 2020-2024, banks and telcos are the top verticals investing in BDA solutions: a report published by the technology research firm IDC.

Capitalizing on financial, transactional, and customer data, the banking industry has always been a haven for big data analytics and will remain so in the foreseeable future. With customers shifting to a more digital experience in banking, bank providers are pushing to scale up their digital transformation as rightfully so. Even though considering complicated legacy banking systems as a hurdle, bank executives recognize prioritizing digital initiatives.

Relatively, the telco industry shows a promising position as the second most prominent investor in big data analytics. Telcos leverage data on customers, sales, and marketing to gain insights for business growth. As we have mentioned before, big data analytics can help optimize the customer experience, which is a big thing for this sector.

State of big data analytics in the Philippines

In Southeast Asia, the Philippines is one of the fastest-growing economies in the region. The country’s expected annual GDP growth surpasses the 5.3% regional average, which stands at a solid 6%. Despite this, competition continues to grow as more digital innovations disrupt traditional business models. Data is everywhere and is becoming more available than ever before. Businesses should see this with a bird’s eye view of how big data can translate to business value. At the same time, customers are engaging with highly personalized experiences.

The Philippines market for big data is still very small. If large firms continue to embrace BDA services and recognize worth, small and medium businesses will follow. This is, however, with a condition. A robust legal framework and support from the government must be in place. Especially, data privacy and security remain to be of major concern.

Optimism in anchoring big data analytics

The big data scene can create around 10 million new jobs in the next five years, Former Globe Telecom chief information officer Henry Aguda shared in an interview with Rappler. If this outlook proves itself, this can directly impact the state of the country’s economy and provide employment opportunities.

Besides employment, possessing hardcore evidence-backed decisions can only do good and be a superpower for firms and organizations. With big data utilized to develop more rigid solutions, businesses can be proactive on what the market wants. In this case, an opportunity is on the rise: tweaking products and services by attending to consumer habits.

Companies worldwide have already established that deeper assimilation and application of data analytics in business translates into greater business value. It is becoming more evident here in the Asia-Pacific region, with enterprises increasing their investment in advanced toots and infrastructure to manage their data better. In this way, companies are taking the right path. A new level of innovation in an increasingly competitive global market.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

4 Proven Ways Data Analytics Increases Revenue

4 Proven Ways Data Analytics Increases Revenue 768 487 Exist Software Labs

With great data analytics comes great returns.

In this day and age, the influx of data has overwhelmed businesses. Which prevents them from maximizing the potential of big data analytics to drive even more revenue and profit. Forbes shares that there are 2.5 quintillion bytes of data created each day but only 0.5% is analyzed.

In the Philippines, only two out of five companies are highly data-driven. But a good thing to note, most Filipino businesses realize this as an advantage. With three out of five having data and analytics (D&A) teams. (PwC Data Analytics Assessment and Readiness Survey, 2020)

We should know by now that making business decisions based on hard evidence from data is becoming a competency and an edge for those who utilize it. Case in point, the McKinsey Global Institute indicates that data-driven organizations are 23 times more likely to acquire customers, six times as likely to retain those customers, and 19 times as likely to be profitable as a result.

Today, getting the most value out of your data is a superpower that business leaders should invest in. It is a profit-increasing ability based on facts, trends, and statistics rather than a gut feel.

In this blog, we’ll share with you four proven ways that embedding a clear data analytics strategy into your business can maximize your returns.

#1. Improving processes to reduce errors

One of the best benefits of having effective data analytics in your business is improving your processes to reduce wasted money and time. 

For instance, you’re in a bread manufacturing company struggling to predict the right numbers to make in a day. In effect, your company overproduces and ends up with wasted unsold food as it expires in less than a week. Besides that, you want your product as fresh as possible when it goes straight to supermarket shelves.

This is where data analytics comes to play. As a result, you can make more accurate forecasting of the demand and supply. Which can significantly reduce the percentage of loss or waste and increase profitability.

#2. Mapping business performance

Another advantage of utilizing data analytics is data visualization. The power to see how your business performs gives you a clearer picture of your goals and targets.

With the right tools to visualize your company data, you’ll be able to map employee performance, team outcomes, company goals, and quarter results, among others. In other words, this empowers your business to look at rooms for improvement and opportunities to consider.

For sure, presenting numbers and graphs has always been a part of your business reviews. That’s why with the right tool, you can also reduce working hours spent on creating manual reports and data consolidation by automating reports generation. Now, your staff can spend valuable time on more critical operational business tasks.

#3. Optimizing the customer experience

Undeniably, the behavior of customers is becoming a lot clearer and available today. You can gather lots of data from various consumer touchpoints. Such as your company website, social media presence, and electronic payments.

After identifying these touchpoints, data analytics can help you optimize the customer experience. Likewise, businesses can harness data to find new customers, track social media interaction with the brand, improve customer retention rate, capture customer inclinations and market trends, predict sales trends, and improve brand experience.

Using the right data can help you plan out an effective marketing strategy and know when to launch your new products and sales. Data is your lens to interact seamlessly with the right customers at the right time.

#4. Getting ahead of the competition

I can’t stress this enough: there’s a multitude of ways your business can utilize your data. So, using this ability to get ahead of your competition is untapped power.

Firstly, data analytics can identify the best channels for your business that offer more significant ROI and focus on them. It will be a driving force to develop a better, informed marketing and sales strategy by analyzing trends and insights from gathered data. You can even avoid spending thousands on marketing campaigns that don’t even address your target audience.

Another one, analyzing data can reveal what your market needs and wants. Keywords can tell you exactly what features your customers are searching. From its frequency, you can infer the relevant priorities of opposing interests. While the relative change can identify and predict trends in customer behavior. It can even understand where customers are struggling the most and how support should be stationed.

Data Analytics empowers Businesses of the Now

Knowing these ways to embed data is just a starting point. If companies can exploit the potential of big data analytics, they can dodge being a business of the past.

Data and analytics are the key forces to an organization’s digital transformation. That’s why Gartner predicts that by 2022, about 90% of corporate strategies will explicitly state data as a significant asset and analytics as a competency.

That’s quite of an outlook. To reiterate: With great data analytics comes great returns.

In a study of over 400 businesses by Bain & Company, they found out that those with the most advanced data analytics get a larger market share. Moreover, they were twice as likely to be in their industry’s top 25% for profitability and five times more likely to make faster decisions than competitors.

Therefore, we can’t ignore the fact that the world’s data production is accelerating. Its effect on the business landscape is getting more and more realized. And being able to derive business decisions from hardcore data analytics is a power that yields better outcomes.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

IoT and Data Integration

Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry Webinar Highlights

Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry Webinar Highlights 768 487 Exist Software Labs

Speaker: Mr. Chris Silerio | VP for Operations

Exist is a software development company that has been in the industry for almost 20 years. Five years ago, we started collaborating and developing systems with companies in the power sector, specifically, generators, distributors, electric cooperatives, and others. We are also one of the technology partners of PEMC and IEMOP. For almost five years, we’ve developed IEMOP’s Central Registration and Settlement System (CRSS) as well as their Trading Operations Central Management Systems. We’ve also recently developed PEMC’s Philippine Renewable Energy Market System (PREMS) through the contract of UNDP and DOE.

We are not an IoT solutions company, but whatever we’re discussing here is based on our vast experience in developing various enterprise system projects in different verticals. That was also made possible by the thorough RnDs and TOCs conducted by our technical team. We are also currently collaborating and partnering with hardware and other software companies that offer IoT devices and solutions.

Agenda:

    1. What is an IoT platform
    2. Discuss the necessary components or services IoT
    3. What are the various ways of implementing an IoT platform
    4. How it could coexist and integrate into a data integration (Data Warehouse/Big Data) platform
    5. How to jump-start your IoT integration journey

What is IoT?

It is the internet connectivity of small devices such as sensors, actuators, smart appliances so they could share and gather information to and from each other.

Some of you might be asking, what’s the big fuss? We already have the internet, and most of you in the power sector already implemented SCADA and PLC before. So how is it any different?

IoT addresses a lot of shortcomings of SCADA and PLC. That is a new level of connectivity between devices. The main difference is when any components within the environment are interconnected, it can act on its own without any human intervention. With all the information it has, it provides rich insight into its entire surrounding environment and instantaneously reacts efficiently by making automatic changes and actions. It can also improve performance and even predict failures, just like Tesla cars or autonomous machines that are in manufacturing plants.

To fully appreciate all the benefits of the internet of things in the energy sector, all of its major components, such as generators, grids, distributors, and consumers, should go smart soon as smart grids cannot function without ‘smart’ consumers.

What is an IoT Platform?

An IoT platform is an integrated service that connects devices and generates its ecosystem online. In the simplest terms, an IoT platform is a group of systems and services where you could operate your entire IoT landscape.

IoT Platform Types, Services, and Offerings

End-to-end IoT Platform

Some companies offer end-to-end solutions. They sell everything – hardware, actual sensors, services, software, and connectivity, plus the device management tools. It is a one-stop-shop, but this tends to be a lot more expensive. You could end up buying features or components that might not apply to your current needs, and yet, you are paying for the entire cost. One of the downsides is that it could become a case of classic single vendor lock-in that most companies steer away from.

Connectivity Management Platform

It is an inherent part of the IoT technology stack. It is the one responsible for connecting and converting data to and from devices and IoT core platforms, regardless if it is on-premise or on-cloud. The primary use case for this type of platform is when you partner with a mobile network to handle communications for security purposes. It means that the data would have to travel through a cellular network rather than flow directly to the internet. But of course, it also introduces new challenges and complexities. So you need to establish your business requirements first.

IoT Edge Platform

We see here the edge devices and the communication gateway. In this type of platform, they provide actual devices and software or tools to manage and configure the devices remotely. With this type, you don’t necessarily need to connect to the cloud as these edge devices are, in a way, functioning already as mini computing and storage machines.

IoT Cloud Platform

It is the most common computing strategy. In this setup, all information from IoT devices could be uploaded to the cloud and a centralized repository. It then runs insights with the aid of full-blown machine learning and artificial intelligence tools. Such simplifies the management of device configurations, managing, and monitoring. Whatever you deploy into the cloud, you could also have on-premise. Your cloud could also push data to the on-premise platform or vice versa.

Edge Computing

An essential part when deciding what type of platform you will choose is whether you would go with edge computing, cloud computing, or hybrid.

Edge computing is bringing the processing at the edge of the local network, not the office LAN, but the network within the autonomous component, for example, a smart car and smart parking garage. These edge devices could send and receive from various IoT devices and communicate directly with other edge devices. 

A case in point is the autonomous Tesla cars that can park itself in an IoT enabled parking garage. The cars going in and out of the garage did not wait for details from the cloud. Instead, they talk to each other directly. Such solved a crucial problem associated with the centralization of data in the cloud. In this model, all processing is distributed, and decision making is down locally by bringing cloud computing capabilities and localizing the data to these devices. It can reduce latency, process data faster, and provide instantaneous insights, as compared to sending everything first to the cloud for analysis.

How Can It Do That?

All the IoT cloud components have been, in a way, miniaturized, streamlined, and deployed in those small edge devices.

Common Components of Edge Devices

Complex Event Processing (CEP) could take data from different IoT devices and act accordingly based on the established patterns. But more often than not, actual CEP models are done on an edge core network. Then, it’s just pushed to the edge devices.

Edge computing devices now support Machine Learning & Artificial Intelligence (ML & AI) locally, but there is always the option of sending the data to the enterprise data warehouse for furth ML & AI processing.

It has the ability to deploy containerized applications as well, either the software package from your solution provider or your own developed applications.

These devices are capable of storage and data management too. These devices now have storage large enough to store all data from different devices, especially in the event of network failure. It is capable of holding data temporarily until the connection gets restored.

When do we consider EDGE over CLOUD?

When your requirements are mission-critical, like, when a machine needs to stop outright to prevent an accident or defective product, sending your data first to the cloud for analysis is not an optimal solution.

Another scenario is when these components are remote, and there is limited or no connectivity to your centralized data center.

Edge computing is still developing, so it has a lot of downsides. Given that the data is localized, security measures might not be as tight as the one securing the cloud security-based systems. It is still a potential privacy and security concern in the edge devices given that the data is stored and already consolidated. That information could also contain sensitive data. Curious people can easily access the device, have the data leaked, or used maliciously.

High availability and failover. Usually, these devices are geographically dispersed. If one edge device goes down, even if you have installed clustered edge devices in one autonomous component, it is still hard to monitor and replace if you have a lot in your IoT plinth.

Scalability. As the number of IoT devices grows, it will be hard to maintain and monitor. Even if you did conduct positive planning, it could only handle so much.

So, choosing between edge and cloud computing depends on your business requirements as well as your business objectives.

Data Integration Platform

The data integration platform is another significant type of platform. Whenever you see an IoT platform presentation, there is always a storage component. It is a combination of several frameworks and tools that will handle the following 

    • Streaming of extracted data from various sources 
    • Routing of data either to your data warehouse or directly to the data consumers’ application
    • Ingestion of data to your data warehouse and OLTP, 
    • Doing a fast filling of data from the database. 

All of these are components of a full-blown enterprise data warehouse or Big Data. What we would like to note is that when you are planning to start your IoT platform, you should first have a data warehouse or Big Data strategy in place.

The goal is to align your IoT storage roadmap with your data warehouse or Big Data roadmap, and not the other way around. Remember that a data warehouse or Big Data setup is also a costly investment and should be treated as a separate project. Having an aligned IoT and data integration strategy could save you a lot of money by maximizing the use of your entire infrastructure.

There are a lot of components that IoT and data warehouse share – streaming, ingestion, and routing. All of these can have its cluster but can be utilized by both your IoT and data platform. With this setup, you can save cost in your infrastructure and reduce the skillset that you need from your IT team.

IoT and Big Data share the same challenges – volume/size of the data coming in, velocity/speed, variety, and veracity/accuracy. When deciding on your IoT storage platform, you have to make sure that it could address all of these challenges.

Here are the few options that you could opt to adopt for your IoT and data warehouse platform.

So, when you have to decide as to which computing strategy or data platform to employ, you need to consider a lot of factors. Don’t just jump on the bandwagon.

Speaker: Mr. Steven Siahetiong | Exist Technical Architect

IoT Components

  1.     Things pertain to sensors, actuators, or any device capable of sending data to the cloud.
  2.     For these devices to send data, it will need to use networking and messaging protocols for the data to be transferred to its destination. Most use wireless access for network connectivity. During the past few years, there have been substantial developments in wireless connectivity protocols. Some of the examples are Bluetooth, Low Power WAN, ZigBee, 6LoWpan, and Thread.
    • Bluetooth is a global 2.4 GHz personal area network for short-range communication. 
    • LPWAN is a type of wireless telecommunication wide area network designed to allow long-range communications at a low bit rate. There are several competing standards and vendors in the LPWAN space. The most prominent of which include Laura and SigFox.
    • ZigBee is a 2.4 GHz mesh local area network lan protocol. It was originally designed for building automation and control, so things like wireless thermostats and lighting systems often use ZigBee.
    • 6LoWpan uses a lightweight IP based communication to travel over lower data rate networks. It is an open IoT network protocol like ZigBee and is primarily used for home and building automation.
    • Thread is an open standard built on IPV6 and 6LoWpan protocols. You could think of it as Google’s version of ZigBee.

For the messaging layer, the most popular are HTTP, MQTT, and CoAP.

    • The CoAp protocol is a client server-based protocol that allows pockets to be shared between client nodes, which are commanded by the CoAP server.
    • The MQTT protocol is communication-based, which is based on the publish/subscribe methodology in which clients receive information through a broker only to the subscribed topic.
  1.     The IoT solution needs to have a platform. An IoT platform combines several IoT functions in one. It can collect and receive your data, convert data between protocols, and store and analyze data. They are available as cloud-based and standalone platforms and are available for many companies, both large and small.

The cloud will have an important role to play in IoT as it will enable companies to create networks, store data, and automate processes without having to build the infrastructure themselves. Such will let IoT services to be developed much quicker and at a lower cost than using traditional in-house systems and services.

As seen on the graph, HTTP involves the largest bandwidth and latency than any other protocols, while CoAP has the least bandwidth and latency.

MQTT offers the highest level of quality of services, with the least interoperability among the four. On the other hand, HTTP was designed for the greatest interoperability on the web and did not include reliability as a core feature.

There are many IoT platforms in the market, and the functionality of these platforms varies enormously. Although all IoT platforms will have dashboards to display data, some platforms are essentially dashboards and are only capable of displaying data from devices. You will often find the terms dashboard and platform used interchangeably.

An IoT dashboard can be considered as a basic IoT platform. A dashboard can usually display data and control devices. However, an IoT platform can usually collect data from various sources, store data, control devices, display data, run tests, deploy device updates, and manage device inventory.

Concepts

Events, insights, and actions are functional concepts that exist across the devices, platforms, and applications of an IoT solution. To further explain, consider an application that monitors the cooling system, temperature for food storage, and calls emergency maintenance services if the temperature becomes dangerously low or high.

The following processes occur in this example:

    • The devices send temperature samples from the primary cooling system to the IoT gateway via the device to cloud events every 30 seconds.
    • These events can generate insights. The IoT platform can evaluate events for any immediate contextual insights, such as temperatures at malfunctioning levels.
    • The generated insights can trigger actions. If the temperature is at a malfunctioning level, the platform can send a command to the backup system to start while the maintenance is en route to the location.

Events represent the device-to-cloud communication in an IoT solution, and maybe notifications, acknowledgments, or telemetry data.

Insights are interpretations of events. It may derive from events directly as contextual insights or transform or stored event data by application event processing for real-time or aggregated insights.

Actions are activities undertaken either programmatically or manually as a device, service, or analog actions.

IoT Customer Scenarios

How can you design an IoT solution to satisfy customer requirements?

    • Manufacturing safety systems must respond to operational data with ultra-low latency and control.
    • Mission-critical systems such as remote mining equipment, connected vessels, or offshore drilling need to analyze and react to data even with limited connectivity to the cloud.
    • The volume of data produced by jet engines or connected cars can be so large that data must be filtered or pre-processed before sending it to the cloud.
    • Regulatory compliance may require some data to be locally anonymized or aggregated before being sent to the cloud.

IoT Edge

IoT devices can connect to the IoT platform directly or through IoT Edge gateways that implement intelligent capabilities. With an IoT edge, you can analyze censored data in near real-time and issue commands when you detect anomalies to stop a machine or trigger alerts. Your streaming logic runs independently off the network connectivity, and you can choose what to send to the cloud for processing or storage. You can also filter or aggregate the data that needs to be sent to the cloud.

The benefits of using edge gateway, particularly in an IoT application, is that moving data processing functions from the cloud to the edge helps ensure accuracy and reliability. Transmitting data to and from the cloud takes time. Even milliseconds are too long for many mission-critical decisions and processes that occur in industrial operations. In the time it takes to send data to and get a response to the cloud, the data could simply become obsolete, resulting in missed opportunities for action, damaged components/products, or risks to the safety of the equipment/personnel.

Edge devices can also send data or commands directly to other devices for immediate action. Edge computing via edge gateway also makes it possible to keep sensitive data on-site to ensure its security. Some devices or systems generate so much data that the bandwidth and resources needed for the cloud to handle this data are too costly. The edge gateway can determine which data can be sent to the cloud and transmit it in the most efficient and usable form.

Industrial Edge LORA 1 Gateway

    • Extremely flexible and powerful
    • Allows to run Lola 1 application and network server locally to setup on-premise private network
    • Aimed to help the transition from legacy to connected automation with its ability of interfacing existing devices.

Types of Edge Gateways

IoT Edge devices can act as communication enablers, local device control systems, and data processors for the IoT cloud platform. IoT Edge devices can run cloud workflows on-premises and can communicate with devices even in offline scenarios.

Cloud gateways can do protocol and identity translation to and from the IoT cloud platform and can execute additional logic on behalf of devices.

In a transparent gateway pattern, the gateway simply passes communications between the devices and the IoT cloud platform.

A protocol translation gateway is also known as an opaque gateway, in contrast with the transparent gateway pattern. In this pattern, devices that do not support MQTT, AMQP, or HTTP can use a gateway device to send data to IoT cloud platforms on their behalf. All information looks like it is coming from one device, the gateway. 

Exist, as a technological solution provider, designed a generic IoT and data integration reference architecture for our customers, especially for the power industry.

The Data Integration layer handles the ingestion of the various data sources. It supports all types of connectors.

IoT Gateway acts as a gateway to connect your devices and accepts data streams using MQTT, CoAP, and HTTP messaging protocols. Data can be ingested as a batch or a real-time stream. With batch processing, data is collected in batches and then fed into an analytics system. A batch is a group of data points collected within a given period. Unlike stream processing, it does not immediately feed data into an analytics system, so results are not available in real-time.

With stream processing, data is fed into an analytics system piece by piece as soon as it is generated. Instead of processing a batch of data overtime, stream processing feeds each data point or a micro-batch directly into an analytics platform. This allows teams to produce key insights in near real-time.

Streaming platforms allow data transformation persistence and allow interactive queries on the streaming application. Kafka is the most popular choice among streaming platforms. 

ETL is the extract, transform, and load process for the data. While the ELT is the extract, load, and transform process. In ETL, data moves from the data source to staging and into the data warehouse. ELT lets the data destination to the transformation, eliminating the need for data staging.

Optionally, you can add a dataflow tool such as NiFi to provide a web UI, which design or control your data pipeline in a graphical representation. NiFi can deal with a great variety of data sources and formats.

For Stream Analytics, the popular choice is Apache Spark Streaming. It is a scalable and fault-tolerant stream processing system that allows data engineers and data scientists to process real-time data from various sources.

Data can be processed using complex algorithms, expressed with high-level functions. Finally, processed data can be pushed out to file systems, databases, and live dashboards. You can apply Spark’s machine learning and graph processing algorithms to data streams.

For the ETL jobs, you can apply data quality steps before loading it to a data warehouse. While for ELT, data is simply dumped into the data lake, and transformations happen on an as-needed basis, and only the data that needs to be analyzed at that time are transformed.

Hot path

    • analyzes data in near-real-time, as it arrives
    • events must be processed with very low latency
    • typically implemented using a stream processing engine
    • the output may trigger an alert or be written in a structured format that can be queried using analytical tools

Warm path

    • holds data that must be available immediately from the device for reporting and visualization

Cold path

    • performs batch processing at longer intervals (hourly or daily)
    • typically operates over large volumes of data, but the results don’t need to be as timely as the hot path.
    • captured and then fed into a batch process

Machine Learning allows predictive algorithms to be executed over historical telemetry data, enabling scenarios such as predictive maintenance. It requires training by telling a machine learning model what it’s trying to predict, similar to how a human child learns.

Speaker: Mr. Jejomar Dimayuga | Exist Technical Architect

Industrial IoT Stack

Sensors

    • Actual things that are located in your physical environment
    • Smart meters or actuators

Edge Devices

    • Can act as the concentrator/aggregator for all of your devices
    • Allow different protocols depending on the support of your devices

Gateway

    • Connectivity to your edge devices

Cloud (or on-premise infrastructure)

    • Where your applications run
    • Offers high availability and high scalability solutions

Insights

    • Where analytics are produced that can be based on historical events, failures, rules, or condition that can communicate back to your device
    • Where algorithms are performed after ingesting and storing the data

Consumption

    • How your data is consumed
    • How can it be displayed via the web, iOS, or Android devices

Security and encryption across the components are must-have. It is a vital requirement for all systems, especially in the IoT solution for energy. Security components such as SSL or HTTPS, LDAP Integration, Single Sign-On, Access Token are already in place.

We accommodate Access Token so we can ensure that your devices are secured in connection to your edge devices and IoT gateway.

We also offer a Single Sign-On authentication to your existing authentication features via the Oauth2.0 protocol.

We can also integrate your authentication via LDAP. We can connect your authentication and your road-based access control to an active directory in your organization if you have one.

To secure your communication layer, we can incorporate SSL or HTTPS with certificates from the latest update of the TLS version.

Another important aspect of any system nowadays is scalability and high availability. In this solution, we can incorporate Docker containerization via the Kubernetes platform. In this way, we can deploy your application or your IoT gateway into multiple replicas to ensure that it is highly available. We can:

    • Reach the maximum daily quota of your required messages
    • Reach the quota of connected devices 
    • Increase the ingestion throughput
    • Increase the processing throughput
    • Set up a DR environment in place to ensure that your operations will be continuous

All of these are in place already so that we can ensure that you will be able to perform your analytics and your business needs. On top of that, we can also integrate into your existing systems, may it be your billing system, billing, or offer system, CRSS, or else.

Speaker: Mr. Chris Silerio | VP for Operations

Things to Consider on Choosing an IoT Platform

    • The capability of your team
    • Geographic coverage
    • Timeline and budget
    • Your IoT and EDW roadmap
    • Your business billing model
    • Scalability of the platform
    • Method of connectivity
    • Integration flexibility
    • Device management
    • Cloud VS. on-premise

Developing your platform in-house is fine as long as you are not pressed by time. Otherwise, it might be better to start with a simple platform and develop on top of it.

Q&A

Q1: With the vast array of open standards for all the layers of IoT, is there a considered safe or conservative stop that anyone can use as a basis or a starter that is easy to pivot towards specific solutions as requirements get clearer? 

A: As shown on the Industrial IoT Stack diagram, we could have the base layer of the devices, followed by the edge devices if you want to go with edge computing. After that, we can set up an IoT gateway on cloud infrastructure.

It also depends on the network or protocol that the sensors, actuators, or your device are supporting. What matters most is that your platform can connect with these various open standards and not limit your entire solution. If there is a new protocol that comes in, your platform should be flexible enough to support these open standards.

 

Q2: What are actuators?

A: Actuators are mostly used in factories. These are devices that are connected to types of machinery. It typically sends telemetry data, like temperature, weight, the roughness of a substance.

 

Q3: What is the experience of Exist in developing IoT systems for energy efficiency applications?

A: We have a couple of projects on the integration of IoT devices, and it is still growing. Our forte is software development integration. But our experiences with these projects gave us enough exposure to the entire ecosystem.

 

Q4: Have you done any IoT applications for use in the Philippine market?

A: We are developing some right now for companies who are just starting their IoT projects. These companies are doing it in phases. We are currently in the first phase. (Company names cannot be disclosed yet.)

 

Q5: Given the poor internet connection in the Philippines, would IoT applications that HTTP use be a challenge?

A: High likely, cloud computing would be a challenge because it will always depend on internet connectivity. In scenarios that it is in remote areas, it would be better to go with edge computing provisioning. That is one of the advantages of an edge over cloud computing. 

Aside from internet connectivity, some providers also offer alternative communication layers, such as cellular networks.

 

Q6: Do you provide a complete solution, like sourcing sensors or edge devices, depending on the requirements?

A: No. There are only a few companies here in the Philippines that offer end-to-end solutions. We are not a hardware company. We are agnostic of software and hardware. We do more on the development of all these layers/components from the IoT devices up to the data integration. But we are partnered with companies that we may utilize to provide an almost end-to-end solution.

 

Q7: For an organization that has mostly in SCADA plus TLC equipment, what would a migration plan look like when moving to a more modern platform that allows a more open choice of technology stack?

A: IoT is not intended to replace SCADA and TLC. It is a different level/layer. Information from SCADA and TLC may be integrated into your IoT platform. We do not have to migrate as we can support SCADA and TLC.

 

Q8: With regards to data storage, would you still recommend Hadoop, or are there better alternatives?

A: We recommend that you go with the storage that can handle multiple parallel processing and the other factors that would support volume processing of data, and speed from streaming data, flowing data from different data sources to your storage devices. There are a lot of databases that support multiple parallel processing, like Greenplum, Oracle, Teradata, and Hadoop.

 

Q9: If I were to engage for an IoT application, what would be the start?

A: It might be better if you could consult first with a third-party company that already has the experience or an offering of IoT solutions in the market. It is quite hard to say outright what the specific software that we are going to use in your IoT platform. We need to undergo some discussions within your organization before we could provide a straightforward platform that you could use. We could always start simple. But regardless of how simple it is, we still need to know what are the factors that we should consider for your organization.

 

Q10: Can you offer an outsource service to support an IoT application?

A: We could somehow handle it, given that we have partners. You may also directly talk to hardware vendors, especially about edge computing devices. We can also give you a list of vendors that you could contact for each IoT component

Data Integration

Data Integration in Power

Data Integration in Power 768 487 Exist Software Labs

“…and when you connect data together, you get power.” 

– Tim Berners-Lee, computer scientist & inventor of the World Wide Web

Way back in 2018, the world is already at a 2.5 quintillion bytes of data per day mark. With the rapid growth in digital transformation and uses of modern-day technologies such as the Internet of Things, the upsurge of this number is simply beyond imagination.

Nowadays, businesses attempt to utilize the collected data and turn it into valuable information to improve their business processes. The process of collecting data has become easier for businesses these days. But the massive outpour of data also posed the risk of data wastage or not knowing how to use it.

In our previous blog, we have mentioned that IoT in power is still on the rise and will keep doing so in the coming years. But most power companies still face their biggest dilemma in dealing with a large amount of data yielded by IoT – data is siloed across different business units.

A data silo is a situation wherein only a selected group can access a particular data set which is isolated from the rest of the organization. The aftermath of having siloed data is problematic for businesses because its non-cohesive nature makes it hard for the teams to view it comprehensively, causing them to miss some useful connections of the collected data. Such also cause the absence of a single source of truth due to data duplication and inconsistent entries under the same account names.

What causes data silos?

According to the Harvard Business Review’s Breaking Down Data Silos, the following causes data silos:

  • Structural – programmers wrote the system application for a particular use or team, and data-sharing is not a requirement in the company
  • Political – there is a sense of proprietorship within the organization that hinders mutual access to the data.
  • Growth – new upgrades are being added to the system, but it is hard to reconcile and integrate the sets of data.
  • Vendor Lock-in – software vendors limit the data access for the users. 

Letting this issue persist in your organization will only make converting data into valuable insights more complex, with much likeliness of being ineffective. That is why combining data from different sources and giving the users a unified view is critical in data analytics. This process is called data integration

In times of the increasing trend in incorporating IoT in different functions of the power sector, dealing with mountains of data have become even complex. As mentioned earlier, the growth, more so, rapid growth, of the system makes it prone to becoming siloed. It is noteworthy to understand the importance of understanding IoT platforms and the various ways of implementing it and how it could coexist and integrate it with data integration platforms. That way, you will be geared up for the boundless changes that modern-day demands and technologies will present.

Join our FREE Webinar!

Join us from Exist, a premier technology, and innovations company, as we hold another learning session entitled Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry.

Data Integration: the Follow through of IoT

Data Integration: the Follow through of IoT 768 487 Exist Software Labs

“In the next century, planet earth will don an electronic skin. It will use the internet as a scaffold to support and transmit its sensations” – Niel Gross, 1999

For decades of harnessing the seemingly limitless power of the internet, we came to a point wherein it became as essential as our basic household needs like electricity. Product creators attempt to incorporate the internet in every device that a person might want or need, making everyday things smarter and more efficient to use.

The ‘Internet of Things’ is connecting everything. It is visible everywhere as if its presence is not as evident yet.

Gartner, Inc. forecasts that the enterprise and automotive Internet of Things (IoT) market will grow to 5.8 billion endpoints in 2020. According to the same study, utilities will be the highest user of IoT endpoints. Senior research director at Gartner Peter Middleton stated that “Electricity smart metering, both residential and commercial, will boost the adoption of IoT among utilities.”

The utility industry started deploying Smart Grid to make meter reading easier. As stated by Middleton as well, IoT devices are also used within power generating plants to monitor equipment over time, to do predictive maintenance, and to provide additional safety oversight.

The implementation of IoT in power and utilities remains steadfast with its overall goal of promoting interoperability and interconnectivity. In this context, specifically, IoT serves as the key to creating energy-driven smart homes. It also means that power and utility companies will strive to tap a range of devices to acquire as much data for them to have real-time visibility into what is happening both to their consumers and the company itself in terms of energy consumption.

Where smart devices are, a storm of data follows. All data HAS to be collected for it to be verified and converted into valuable insights.

The tie between IoT and data does not end when the data is collected. It then becomes the responsibility of the companies to create value out of this massive amount of data to improve business outcomes and remain relevant in the long run. Here is where data integration comes into play. It enables businesses to prevent data silos and consolidate data from different sources into a single, unified view.

Source: 

Join our FREE Webinar!

Join us from Exist, a premier technology, and innovations company, as we hold another learning session entitled Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry.

Does Your Organization Have the Pivotal X-Factor?

Does Your Organization Have the Pivotal X-Factor? 768 487 Exist Software Labs

“No, you can’t always get what you want. You can’t always get what you want. You can’t always get what you want.”

Thus echoed the famous Rolling Stones tune of 1969. The principle may be true in many cases, but when it comes to the Data Integration needed for turning up actionable business insights, getting at all the Data Sources that you want can actually be an “always” kinda thing. Let me explain.

What is Data Integration?

To put it simply, you want access to all the data that are relevant to your organization—data that will enable you to do the analytics that drive business-building decisions. Having access to the data, or Data Integration, can mean two things:

    • You haul the data off from various data sources into a central data hub, or…
    • You directly interface with the data in-place (where they reside) from a data access platform

The first involves your classic ETL (Extract-Transform-Load) paradigm, and may also include ELT (Extract-Load-Transform), mostly used in dealing with unstructured data, and real-time, event-driven streaming. The second is what’s become a buzzword nowadays called, Data Federation or Data Virtualization.

What if I told you that you don’t have to buy separate Data Integration tools to accomplish these two ways of getting at data? What if I told you that all you need is the X-Factor…the Pivotal X-Factor!

What is Greenplum PXF?

A primary feature of the Greenplum Modern Data Analytics Platform, one that makes it a cut above the rest, is the Greenplum Platform Extension Framework (PXF). Greenplum PXF provides data connectors to all relational data sources supported by JDBC. It also has HDFS, Hive, and HBase connectors to all major commercial Hadoop distributions like Cloudera, Hortonworks Data Platform, and MapR, along with generic Apache Hadoop distributions. Is your data up on Cloud object stores? PXF provides connectors to Text, Avro, JSON, Parquet, AvroSequenceFile, and SequenceFile data on Azure, Google Cloud, Minio, and S3 object stores as well.

By defining External Tables that point to these data sources, you are able to retrieve and modify external data from the comforts of your own Greenplum home, as it were, by merely issuing SQL statements that behave as if the tables were actually local. And given Greenplum’s MPP architecture, access to these outside-the-fence data is lightning fast! How does inserting 71 million+ records from SQL Server into a 3-segment Greenplum cluster in 13 to 18 minutes sound?!

A Simple Example

Let’s say we have data that we want to do analytics on residing in two different kinds of databases, one SQL Server and the other Postgres. And suppose some important information also resides in CSV files provided by some department.

The traditional way of acquiring the data from these three different data sources is to use a separate ETL tool to define jobs that would physically transfer the data to destination tables. While this is OK, it leaves you committed to the data that you have already ingested and any mixing and matching of relationships can only be done after the fact and at the cost of physically transferring data.

What if you wanted the flexibility of first querying the data in these distinct data sources individually, and then together, establishing relationships between your SQL Server, Postgres, and CSV data without needing to load them physically first into your data repository? 

 With Greenplum PXF (the Pivotal X-Factor!), you can do just that!

So these are your three different data sources:

After configuring PXF JDBC settings under the hood, you define External Tables to the particular tables in SQL Server and Postgres that you want to access:

SQL Server External Table (PXF)

CREATE EXTERNAL TABLE pxf_sqlserver_TestTable
(
      ID int,
      Name varchar(50),
      Source varchar(20)
)
LOCATION ('pxf://DICT_Test.dbo.TestTable?PROFILE=Jdbc&SERVER=sqlserver')
FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');

Postgres External Table (PXF)

CREATE EXTERNAL TABLE pxf_postgres_dimairline 
(
      airlineid smallint,
      airlinename varchar(95)
)
LOCATION ('pxf://public.dimairline?PROFILE=Jdbc&SERVER=postgres')
FORMAT ‘CUSTOM' (FORMATTER='pxfwritable_import');

For accessing the CSV file, we also define an External Table but with the use of another awesome Greenplum innovation called gpfdist (which will be the subject of another blog):

CSVExternal Table (gpfdist)

CREATE EXTERNAL TABLE ext_csv_address
(
      id int, 
      address varchar(100),
      status varchar(10))
LOCATION ('gpfdist://gpmdw:8080/*.csv')
FORMAT 'CSV' ( DELIMITER ',' );

And with just one query, you can retrieve the data from all three data sources, relating them to each other as you please:

SELECT * FROM
pxf_sqlserver_TestTable A → SQL Server
RIGHT JOIN pxf_postgres_dimairline B → Postgres
      ON A.id = B.airlineid
LEFT JOIN ext_csv_address C → CSV file
      ON B.airlineid = C.id
ORDER BY A.id LIMIT 10;


This is the Data Federation or Data Virtualization use case. You can also use the External Tables as a simple way to implement ETL by inserting to your pre-defined Greenplum tables from them, like so:

INSERT INTO gp_sqlserver_TestTable
SELECT * FROM pxf_sqlserver_TestTable;

Take note that this insertion to Greenplum is done in parallel across all segment hosts, thereby giving it lightning-fast load performance (71+M recs in 13 to 18 minutes!)

If transformations on the data are required, then creating user-defined functions in Greenplum should do the trick.

Parting Words

So there you have it. With Greenplum PXF, you can always get at the data that you want, enabling your organization to transform data into actionable insights fast and quick. You do want your business to stick around as long as the Rolling Stones, don’t you? 😉