Data Solutions

Exist Software Labs Inc, Informatica Data Governance Pocket Session

Exist Software Labs Inc and Informatica Pocket Session 2022: Realizing Data Governance Benefits in a Cloud-Hybrid World

Exist Software Labs Inc and Informatica Pocket Session 2022: Realizing Data Governance Benefits in a Cloud-Hybrid World 650 486 Exist Software Labs

Exist Software Labs Inc and Informatica Pocket Session: Realizing Data Governance Benefits in a Cloud-Hybrid World

On September 15, Exist Software Labs, in a joint effort with Informatica, gathered various market leaders from various verticals to conduct another pocket session on Data Governance and its benefits in a Cloud-Hybrid World.

Exist and Informatica: Realizing Data Governance Benefits in a Cloud-Hybrid World

Jon Teo, Data Governance and Privacy Expert at APJ spoke at the event about its benefits. He demonstrates how it helped various industries such as healthcare, automotive, insurance, manufacturing, power, and others around the world by leveraging its risk and compliance to protect the enterprise, as well as data intelligence that unlocks more value and data opportunities for businesses.

According to him, rapid cloud adaptation and a hybrid ecosystem generate more volume from more sources, making it difficult to discover, manage, and control data, requiring the urgent need for an agile governance approach.

Kingsley Dsouza, a Technical Data Governance Privacy Domain Expert, was one of the speakers who also demonstrated Informatica’s services. According to him, “Data Governance platform helps users in finding information that will assist them in solving their day-to-day business problems, which most organizations struggle with and take a long time to process.”

It’s no secret that the Asia-Pacific region lags behind the rest of the world in data management, with less than 50% of organizations having standardized data management capabilities. As the amount of data generated in the region continues to grow at an exponential rate, organizations are scrambling to find effective ways to manage and store all of this information, which is where the agile governance approach comes into play.

Mitigate security risks and ensure compliance with data privacy laws by standardizing your data management! Get in touch with our team to know more.

Download our FREE DATASHEET!

Begin your journey toward data maturity.
and transform into a data-driven organization today!

Did you miss the event?

Watch the Realizing Data Governance Benefits in a Cloud-Hybrid World Video On Demand now!

Exist Software Labs Inc. and Informatica

Exist Software Labs Inc. and Informatica held a joint Pocket Session on Intelligent Data Management Cloud at the Shangri-La Fort Hotel in BGC!

Exist Software Labs Inc. and Informatica held a joint Pocket Session on Intelligent Data Management Cloud at the Shangri-La Fort Hotel in BGC! 650 486 Exist Software Labs

Exist Software Labs Inc. and Informatica held a joint Pocket Session on Intelligent Data Management Cloud at the Shangri-La Fort Hotel in BGC!

‘Data is the new oil. Like oil, data is valuable, but if unrefined it cannot really be used. It has to be managed/processed (integrated, mapped, transformed) to create a valuable entity which provides insights that drives profitable activities.’ – Informatica

A collaboration with Informatica


Exist Software Labs inc collaborated with Informatica for an exclusive face-to-face event last July 28, 2022, at the Shangri-La Fort Hotel in BGC. The guests were able to meet with data management expert and Informatica’s Head of Cloud Product Specialist, Daniel Hein, who shared how companies can bridge the gap between technology and business through automation, integration, and data governance, unlocking true business value from data.

 

The world is changing, and so are your business’s needs. You must be able to adapt quickly to keep up with the changes. “In the last two years, a lot has changed. We are faced with new ways of doing business; the world is moving to a data-driven digital economy… However, there are CONSTRAINTS that you must overcome.” says Daniel Hein, Head of Cloud Product Specialists, APAC and Japan.

That is why businesses must change their approach. The new Intelligent Data Management Cloud intends to help clients with that! The first and most comprehensive AI-powered data management solution in the industry. A single cloud platform. Every cloud-native service you’ll ever need for next-generation data management.

IDMC

Meet the new Intelligent Data Management Cloud of Informatica!

IDMC platform cuts through red tape and provides accurate AI models across your organization so you can make timely decisions based on the most up-to-date information.

It also gives you 360-degree views of your data across all areas of your business—so you can see who has access and what they’re doing with it—and allows easy workflow management.

It is built on top of an enterprise cloud platform; and is equipped with a powerful security model that helps keep sensitive information secure from hackers.

If you’re looking for a way to help your company prepare for this transition and stay competitive in an ever-changing marketplace, look no further! We specialize in helping companies not only to keep pace but also to improve their bottom line through digital transformation.

Download our FREE DATASHEET!

Begin your journey toward data maturity.
and transform into a data-driven organization today!

YugabyteDB

YugabyteDB in 2022: What Is Good For?

YugabyteDB in 2022: What Is Good For? 800 507 Exist Software Labs

Why YugabyteDB?

The database evolved from the network & hierarchical model of the 60s to what eventually became the dominant force in databases in the 70s up to the present time—the relational model.

From this model arose the big names in Relational Database Management Systems like DB2, Oracle, and Sybase. In the 80s, a player emerged that would forever change the RDBMS landscape: Postgres.

Postgres is the only RDBMS to have won DB of the year 3 times: 2017, 2018, & 2020. This speaks of the universal trust that developers have placed in the system and this has only increased with the rise of open-source software and cloud-native computing.

Which begs the question: What is cloud-native computing?

To put it simply, the rise in the services-based model championed by the major cloud providers like AWS, GCP, & Azure has given way to a new way of developing applications, with key emphases on scalability, resilience, high availability, and agile deployments.

Combining the relational supremacy of Postgres and cloud-native computing, we are then faced with the next step in the evolution of the RDBMS—Distributed SQL— and YugabyteDB is the No. 1 Distributed SQL platform today.

Use Cases

What is YugabyteDB good for?

I will mention 4: cloud-native applications, applications requiring massive scale, geo-distributed workloads, and traditional RDBMS modernization.

1. Cloud-native applications – Build stateful microservices by leveraging modern cloud-native frameworks such as GraphQL, or in any language of your choice like Django, Spring, Go, etc.

2. Massive-scale applications – Seamlessly deploy IoT and streaming services that demand high throughput, support for large data sets, and/or many concurrent connections.

3. Geo-distribution – Move data closer to users for resilience, performance, and compliance with the industry’s largest choice of deployment and replication models.

4. RDBMS modernization – Evolve Oracle, SQL Server, and DB2 workloads to a modern distributed SQL database with minimal disruption. If you can migrate to PostgreSQL, you can migrate to YugabyteDB.


Real-World Projects

We have deployed YugabyteDB in several government and educational institutions that employed cloud-native applications development—requiring user connectivity from various geo-locations—along with the migration of MySQL databases.

YugabyteDB has delivered in all of these projects, requiring very minimal tweaking in existing SQL code and even speeding up queries in many instances.

YugabyteDB will very soon be the de facto RDBMS of choice in a cloud-native world.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with our Data Solutions Services. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

Energy Tech Trends, Java, Java Philippines

2022 Energy Tech Trends to watch

2022 Energy Tech Trends to watch 800 507 Exist Software Labs

The COVID-19 pandemic, numerous heavy typhoons, and other unfortunate events affect all business sectors in the country. One of the major industries affected is Energy and Utilities, and it highlighted the necessity for The Energy Industry to adopt a sustainable perspective and improve the technology system.

Energy Tech Trends 2022

With the rising demand for energy in 2022, the Department of Energy will continue its advocacy to produce renewable energy to cut market prices and achieve sustainability. On the other hand, the private sector will continue to develop technology systems to achieve efficiency, and high effectiveness and keep up with the continuously evolving energy sector.

While numerous technology solutions can assist these firms in achieving their digital objectives, only a few are expected to have a significant influence in 2022. These are the 2022 Energy Tech Trends to watch!

1. Powering Digital Economy Through IoT (Internet of Things)

IoT, or the Internet of Things, has played a vital role in advancing digitization in several industries, including IT, energy, agriculture, healthcare, and many more.

It is one of the most advanced technologies, and one of its advantages is that it improves the efficiency of several businesses, including energy. And as for the energy sector, one of the most important functions of IoT is energy conservation. 

The Internet of Things enables electricity firms to read data in real-time. It enables them to quickly gather, calculate, and analyze data to improve decision-making. It also helps the energy industry transform into an integrated system, resulting in a smart solution that is equipped with advanced technology to increase industry value and maintain asset efficiency for the benefit of the economy.

2. Fifth-Generation Technology will establish the connection

Many companies will continue to improve their systems in 2022. The sector will continue to advance the electric grid to make it more reliable and less expensive, thanks to the national government’s directive to push for renewable energy and industry’s developing market demands.  

These companies rely on Millions of connected devices and digital systems, such as smart meters, sensors, management systems, to communicate data from many locations. And with their objective to digitize their system, they need to have fast and dependable technology, thus creating a need to access 5G technology. 

5G technology is the next generation of cellular technology after 4G. It has faster speed, lower latency, and the capacity to connect more devices at the same time.

Fifth-generation wireless technology will provide new features and more efficient smart grids. New 5G mobile networks will assist the integration of unconnected devices into new smart grids; it will also help the development of new electricity load forecasting software for accurate energy monitoring and forecasting. Organizations will now be able to receive and process the massive volume of data at quicker speeds with no chance of downtime.

3. Companies in the energy industry will continue to migrate to the cloud

The cloud holds the potential for endless growth, system efficiencies, and digital integration in any business industry.

With the Power industry’s continued growth, it needs a system that can handle its complex process and massive data efficiently, effectively, and precisely; this is where the cloud can help.

The cloud has the potential to change every aspect of the energy value chain. Connectivity, scalability, analytics, and automation can all help you save money and increase profits in countless ways.

Thanks to Exist’s strong foundation in the power industry, with its business solutions to industry market leaders and cloud services to other business verticals, we can now quickly apply tailored advancements to your company.

4. Artificial Intelligence will revolutionize the game.

Artificial Intelligence (AI) is becoming relevant in the energy industry and has great potential for future energy system structures.

Digital technologies such as Artificial Intelligence (AI) will make energy sector systems more intelligent, efficient, dependable, and sustainable, which benefits the entire energy sector chain, from generation to transmission to distribution to the consumer.

In terms of alignment with the government’s ambitions, AI would also benefit Renewable Energy. With the growing use of renewable energy sources, it is becoming increasingly difficult to regulate the megawatts that are fed into the grid; with this, power networks will be unstable and prone to blackouts.

With this technology, renewable energy sources may now provide real-time, accurate data that allows AI to predict capacity levels.

5. Energy Sector will embrace the power of Machine Learning

With Machine Learning (ML), it’s as if you have a sophisticated human mind monitoring your system, complete with advanced self-learning algorithms, taking your data to a whole new level by making human-like decisions based on current AI data.

ML employs approaches that can be applied to predictive maintenance. Power lines, machines, and stations, in essence, are outfitted with sensors that capture operating time series data.

With enough data, your system can now forecast if a failure will occur in your system, allowing you to more efficiently monitor maintenance, reduce downtime, and avert system failure as soon as possible; thus, lowering your system expenditures.

6. Taking Advantage of Big Data

It’s one thing to get your data, but it’s quite another to use it to your advantage. 

Big Data Analytics has the potential to be a key driver in achieving optimal company performance in the energy sector.

Big data can help the energy business in many ways, including improved supply chain management, enhanced customer satisfaction, optimizing business efficiency, analyzing future risks and possibilities, and much more.

As a result, more and more energy companies are becoming more competitive. A superior business strategy that incorporates a large amount of data and efficient processes is assisting them in developing company value and increasing customer satisfaction.

So, how can you achieve the advanced system and follow Energy Tech Trends?

Whether you like it or not, the energy sector will continue to advance its technological innovation.

In this regard, it is important to look for an innovative partner who can add value to your organization, and this is where Exist Software Labs can assist you! With our extensive experience in the energy industry, we could bring you the innovation you deserve.

Empower your system today!

Learn how to fully automate your processes to create a more competitive, transparent, and efficient system.
Take your power system to the next level!

Exist Data Solutions: The Elephant Behind the Excellence. Java, Java Philippines

Exist Data Solutions 2022: The Elephant Behind the Excellence

Exist Data Solutions 2022: The Elephant Behind the Excellence 650 486 Exist Software Labs

The preceding year, 2021, was an eventful year for EXIST Data Solutions: new team members were added, new technologies were learned, and new projects were implemented.

In the enterprise database front, PostgrEX was implemented in a prestigious 5-star hotel and casino, a state university, and a major security agency handling the biggest mall in the country.

YugabyteDB, the No. 1 cloud-native, distributed SQL database in the world, was implemented in 3 government projects, and premium EnterpriseDB support was rendered to the country’s primary energy market corporation.

On the Exist data solutions front, Greenplum was also successfully implemented in 3 government projects, thereby enabling these entities to turn their data into actionable insights.

But what do all these business-transforming technologies have in common? In a word: Postgres.

Postgres is the database engine upon which PostgrEX, YugabyteDB, EDB, and Greenplum are based. With most of them, modifications in varying degrees were done to core Postgres to deliver a product that is still Postgres, but better!

As indicated in the article, Databases in 2021: A Year in Review, the dominance of Postgres in the year 2021 was undeniable:

The conventional wisdom among developers has shifted: PostgreSQL has become the first choice in new applications. It is reliable. It has many features and keeps adding more.

In 2010, the PostgreSQL development team switched to a more aggressive release schedule to put out a new major version once per year (H/T Tomas Vondra). And of course, PostgreSQL is open-source.

PostgreSQL compatibility is a distinguishing feature for a lot of systems now.

Such compatibility is achieved by supporting PostgreSQL’s SQL dialect (DuckDB), wire protocol (QuestDB, HyPer), or the entire front-end (Amazon Aurora, YugaByte, Yellowbrick). The big players have jumped on board.

Google announced in October that they added PostgreSQL compatibility in Cloud Spanner. Also in October, Amazon announced the Babelfish feature for converting SQL Server queries into Aurora PostgreSQL.

One measurement of the popularity of a database is the DB-Engine rankings. This ranking is not perfect and the score is somewhat subjective, but it’s a reasonable approximation for the top 10 systems.

As of December 2021, the ranking shows that while PostgreSQL remains the fourth most popular database (after Oracle, MySQL, and MSSQL), it reduced the gap with MSSQL in the past year.

Another trend to consider is how often PostgreSQL and Exist Data Solutions is mentioned in online communities. This gives another signal for what people are talking about in databases.

What does all this mean for you and your business? It means you can entrust your most mission-critical applications to Exist Data Solutions, Postgres, and its derivatives.

It means you can break free of vendor lock-in and redirect cost savings to core business initiatives. It means your company can be a better version of itself–a more profitable version–in the year 2022!

Be a Data-Driven Organization.

An organization that is data-driven recognizes the value of data and bases decisions on factual information. This organization has invested time and money to acquire Data Solutions Services that can source data from both inside and outside the company.

If your organization is like the majority, you’re seeking methods to accomplish more with less. However, you don’t want to spend a fortune to get the information you require.

When you need data analytics services or an enterprise-grade database, Exist Software Labs Inc.’s Data Solutions services can open up new possibilities for you.

Contact us and find out how EXIST Data Solutions can meet all your database-related requirements.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

A Fully Dockerized MySQL to YugabyteDB Migration Strategy Using pgloader, Java, Java Philippines

A Fully Dockerized MySQL to YugabyteDB Migration Strategy in 2022 Using pgloader

A Fully Dockerized MySQL to YugabyteDB Migration Strategy in 2022 Using pgloader 768 487 Exist Software Labs

YugabyteDB Migration Strategy


While there have been many who began their journey to relational databases with the simple and popular MySQL, the evolution of business use cases involving more than read optimization and the need for more performant, full-fledged, read/write-optimized OLTP systems have given rise to a widespread migration from MySQL to Postgres.

Along with this, the transition from monolithic to cloud-native has also paved the way for distributed SQL systems that allow for read/write functionality in every node of the database cluster (while maintaining ACID-compliance across all nodes) and cloud-agnostic deployments of these nodes across geographic zones and regions. This is the future of the database, a future where reliability, accessibility, and scalability are built into the product. The future of the database is YugabyteDB.
 

From MySQL to YugabyteDBfast!

The method that we will be using to migrate a MySQL database to YugabyteDB is through the use of pgloader, a very reliable tool for migrating from MySQL (even SQL Server) to Postgres. We will first migrate the MySQL database to a Dockerized Postgres instance using Dockerized pgloader.

Once the MySQL database has been migrated to Postgres, we will then use the ysql_dump utility that comes with every installation of YugabyteDB to dump the Postgres database into a YugabyteDB-friendly format. This is one of the very useful traits of ysql_dump: it ensures that your Postgres dump can be fully restored in a YugabyteDB instance.

After getting the dump, we will restore this dump in the blank YugabyteDB database that we’ve created beforehand, thereby completing the migration from MySQL to YugabyteDB!

 

Steps

1. Get the Postgres Docker container

docker run -e POSTGRES_HOST_AUTH_METHOD=trust -p 5432:5432 -d postgres:11

2. Create the MySQL database counterpart in Dockerized Postgres

CREATE DATABASE <db name>;

3. Run Dockerized pgloader to load from MySQL to Dockerized Postgres

docker run --rm --name pgloader dimitri/pgloader:latest pgloader --debug mysql://<user name>:<password>@<ip address of MySQL DB server>:3306/<source database name> postgresql://postgres@<ip address of Dockerized Postgres>:5432/<destination database name>

*If a user error is encountered, make sure the user and IP address combination indicated in the error is created in the MySQL source and has access to the databases to be migrated.”

4. Since pgloader creates a Postgres schema using the database name and puts the tables there, we can change the schema name to “public”

DO LANGUAGE plpgsql
     $body$
     DECLARE
     l_old_schema NAME = '<schema name>';
     l_new_schema NAME = 'public';
     l_sql TEXT;
     BEGIN
     FOR l_sql IN
     SELECT
          format('ALTER TABLE %I.%I SET SCHEMA %I', n.nspname, c.relname, l_new_schema)
     FROM pg_class c
          JOIN pg_namespace n ON n.oid = c.relnamespace
     WHERE
     n.nspname = l_old_schema AND
     c.relkind = 'r'
     LOOP
     RAISE NOTICE 'applying %', l_sql;
     EXECUTE l_sql;
     END LOOP;
     END;
     $body$;

5. In this example, we will be using Dockerized Yugabyte as the destination (also applies to other form factors)

a. 1-node cluster with no persistence: 

docker run -d --name yugabyte  -p7000:7000 -p9000:9000 -p5433:5433 -p9042:9042 yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false

b. With persistence:

docker run -d --name yugabyte  -p7000:7000 -p9000:9000 -p5433:5433 -p9042:9042 -v ~/yb_data:/home/yugabyte/var yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false

6. Go inside the Yugabyte container

a. To access the interactive terminal of the container:

docker exec -it <yugabyte container id> /bin/bash

b. Go to the bin directory:

cd /home/yugabyte/postgres/bin

c. Make sure destination database exists in YugabyteDB:

CREATE DATABASE <destination yugabytedb name>;

d. Dump the database in the Postgres container:

./ysql_dump -h <ip address of Postgres container> -U postgres -d <database name of postgres db> -p 5432 -f <dump name>.sql

e. Restore the Postgres dump in the blank database in the YugabyteDB instance:

./ysqlsh -p 5433 -d <database name of destination yugabyte db> -f <dump name>.sql

 

And there you have it! You have successfully migrated your MySQL database to the future of the database. You have migrated to YugabyteDB!

yugabytedb migration

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

The Future of the Database: YugabyteDB, Java, Java Philippines

The Future of the Database: YugabyteDB

The Future of the Database: YugabyteDB 768 487 Exist Software Labs

The Future of the Database

The journey to application modernization brought about by the cloud-native renaissance continues, and the benefits to be had are truly being enjoyed by the enterprises that embrace the path. Speed, scalability, resiliency, and agility may seem to just be industry buzzwords, but in reality, they translate to better application deployment, performance, and availability, which further translate to what really matters: happy customers.

This has given way to the concomitant need for databases to adapt to this need for speed, scalability, resiliency, and agility. The way traditional databases have implemented a single-node access to the database cluster via the master node has proven untenable in a commercial environment wherein the need to scale users, not just locally, but across the regional and geographical divide, has become dire and ubiquitous.

This is where the gap is filled by YugabyteDB.

 

What is YugabyteDB?

What is YugabyteDB?

YugabyteDB is a transactional, distributed SQL database that was designed primarily to possess the virtues of the cloud-native philosophy. Its creators wanted a chiefly OLTP database that was fast, easy to add more nodes to, able to tolerate node failures, upgradable without incurring any downtime, and deployable in all form factors (public/private cloud, VMs, and on-prem).

Being a distributed SQL database, it has automatic distribution of data across nodes in a cluster, automatic replication of data in a strongly consistent manner, support for distributed query execution so clients do not need to know about the underlying distribution of data, and support for distributed ACID transactions.

It is a multi-API database that exposes the following APIs (more will be added in the future): 

  • YSQL – an ANSI SQL, fully-relational API that is completely compatible with PostgreSQL 11.2
  • YCQL – a semi-relational SQL API that is based on the Cassandra Query Language

It is a Consistent and Partition Tolerant (CP) database in that in the event of a network partition within the database cluster wherein one of the nodes cannot communicate with the other nodes and determine majority membership, data consistency over availability is prioritized by the system and this node will not be able to accept writes, whereas the nodes that are still part of the majority will remain unaffected.

It is completely open source, released under the Apache 2.0 license.

 

What are the key benefits of YugabyteDB?

The following are some of the benefits that are immediately enjoyed “out-of-the-box”:

  • No single point of failure given all nodes are equal
  • Distributed transactions across any number of nodes
  • Scale write throughput linearly across multiple nodes and/or geographic regions.
  • Low-latency reads and high-throughput writes.
  • Strongly consistent, zero data loss writes.
  • Cloud-neutral deployments with a Kubernetes-native database.
  • Automatic failover and native repair.
  • 100% Apache 2.0 open source even for enterprise features.

In other words, you get a cloud-native, transactional, distributed SQL database system that allows you to read and write on every node in the cluster (with ACID assurance), distribute your application load across many nodes in many regions and geographies, read and write data fast, deploy anywhere, and be highly available—all in open source!

 

Use Cases

YugabyteDB is perfect for:Use Cases of YugabyteDB

Just this morning, social media personality, James Deakin, posted on his FB wall about a particular bank whose “app feels like it’s running on windows 95” (his own words). He ended up closing his account due to the overall poor customer experience brought on by the subpar performance of this bank’s client-facing, internet applications, along with other concerns.

YugabyteDB is perfect for the client-facing, Internet, transactional application.

Want to know more about the Yuggernaut of Distributed SQL? Contact us.

Exist is your data solutions partner of choice!

Explore the next level of your digital transformation journey with big data and analytics. Let’s look at opportunities to better maximize your ROI by turning your data into actionable intelligence. Connect with us today, and we’ll proudly collaborate with you!

Befriending Your Data in 2021, Java, Java Philippines

Befriending Your eye-opening Data in 2021

Befriending Your eye-opening Data in 2021 768 487 Exist Software Labs

It’s the new year and everybody is still living in the wake of the COVID-19 pandemic. We all need a friend in times of trouble and this is no different in the case of business organizations.

This year, 2021, the friend that your company needs more than ever, especially in these trying times, is data.

Given the disruption that this virus caused in the preceding year, enterprises need to start (if they haven’t already) befriending their own internal data, and perhaps external data as well if they are to at least stay viable and at most grow.

The following are some insights from respected data management leaders on how to make friends with your data this year:

  • “Data warehouses are not going to disappear. Data warehouses will continue to be an important legacy technology that organizations will use for mission-critical business applications well into the future.

    With the transition to the cloud, data warehouses got a fresh new look and offer some modern attractive capabilities including self-service and serverless.

    With the rise of the cloud, data lakes are the new kid on the block. Data lakes are becoming a commodity, a legacy technology in their own right. Their rapid emergence from the innovation stage means two things going forward.

    First, organizations will demand simpler, easier to manage, and more cost-effective means of extracting usable business intelligence from their data lakes, using as many data sources as possible.

    Second, those same organizations will want the above benefit to be delivered via tools that do not lock them into proprietary data management platforms.

    In short, 2021 will begin to see the rapid introduction and evolution of tools that allow users to keep their data lakes in one place and under their control while driving performance up and cost down.”

  • “Distributed analytical databases and affordable scalable storage are merging into a single new thing called either a unified analytics warehouse or a data lake house depending on who you’re talking to.

    Data lake vendors are scrambling to add ACID capabilities, improve SQL performance, add governance, resource management, security, lineage, and all the things that data warehouse vendors have been perfecting for the last three or four decades.

    During the ten years, while data lake software has been coalescing, analytical databases have seen their benefits and added them to their existing stacks: unlimited scale, support for widely varied data types, fast ingestion of streaming data, schema-on-read, and machine learning capabilities.

    Just like a lot of things used to claim to be cloudy before they really were, some vendors will claim to be a unified analytics warehouse when they’ve just jammed the two architectures together into a complicated mess, but everyone is racing to make it happen for real.

    I think the data warehouse vendors have an unbeatable head start because building a solid, dependable analytical database like Vertica can take ten years or more alone.

    The data lake vendors have only been around about ten years, and are scrambling to play catch-up.”

  • “One single SQL query for all data workloads

    The way forward is based not only on automation but also on how quickly and widely you can make your analytics accessible and shareable.

    Analytics gives you a clear direction of what your next steps should be to keep customers and employees happy, and even save lives. Managing your data is no longer a luxury, but a necessity–and determines how successful you or your company will be.

    If you can remove the complexity or cost of managing data, you’ll be very effective.

    Ultimately, the winner of the space will take the complexity and cost out of data management, and workloads will be unified so you can write one single SQL query to manage and access all workloads across multiple data residencies.”

  • “Expect more enterprises to declare the battle between data lakes and data warehouses over in 2021 – and focus on driving outcomes and modernizing.

    Data warehouses can continue to support reporting and business intelligence, while modern cloud data lakes support all analytics, AI and ML enablement far more flexibly, scalably, and inexpensively than ever – so enterprises can go transform quickly.

    Cloud migrations and related cloud data lake implementations will get demonstrably faster and easier as DIY approaches are replaced by turnkey SaaS platforms.

    Such solutions will slash production cloud data lake deployment times from months to minutes while controlling costs and providing the continuous operations, security and compliance, AI and ML enablement, and self-service access required for modern analytics initiatives.

    That means that migrations that used to take 9-12+ months are complete in a fraction of the time.”

  • “Co-locating analytics and operational data results in faster data processing to accelerate actionable insights and response times for time-sensitive applications such as dynamic pricing, hyper-personalized recommendations, real-time fraud and risk analysis, business process optimization, predictive maintenance, and more.

    To successfully deploy analytics and ML in production, a more efficient Data Architecture will be deployed, combining OLTP (CRM, ERP, billing, etc.) with OLAP (data lake, data warehouse, BI, etc.) systems with the ability to build the feature vector more quickly, and with more data for accurate, timely results.”

To summarize the various points made by these industry pundits:

1

SQL-driven data warehouses are here to stay and will continue to be the data analytics platform of choice for enterprises in the current year.

2

Data management platforms that integrate well with existing data lakes will dominate as opposed to platforms that focus on one or the other.

3

Data management platforms that have built-in AI/ML functionalities will dominate as well, as this eliminates the cost and complexity of separate AI/ML analytics platforms.

4

Data management platforms that are cloud-ready will also have an edge over those that are not.

Is there a data management platform that possesses all these qualities and has a proven track record in Fortune 500 companies?

Yes, there is. It’s called Greenplum. Read about it here.

IoT and Data Integration. Java Philippines. Java.

Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry Webinar Highlights (2020)

Unlocking the Power of IoT and Data Integration to Boost Transformation in the Energy Industry Webinar Highlights (2020) 768 487 Exist Software Labs

Speaker: Mr. Chris Silerio | VP for Operations

Exist is a software development company that has been in the industry for almost 20 years. Five years ago, we started collaborating and developing systems with companies in the power sector, specifically, generators, distributors, electric cooperatives, and others. We are also one of the technology partners of PEMC and IEMOP. For almost five years, we’ve developed IEMOP’s Central Registration and Settlement System (CRSS) as well as their Trading Operations Central Management Systems. We’ve also recently developed PEMC’s Philippine Renewable Energy Market System (PREMS) through the contract of UNDP and DOE.

We are not an IoT solutions company, but whatever we’re discussing here is based on our vast experience in developing various enterprise system projects in different verticals. That was also made possible by the thorough RnDs and TOCs conducted by our technical team. We are also currently collaborating and partnering with hardware and other software companies that offer IoT devices and solutions.

Agenda:

    1. What is an IoT platform
    2. Discuss the necessary components or services IoT
    3. What are the various ways of implementing an IoT platform
    4. How it could coexist and integrate into a data integration (Data Warehouse/Big Data) platform
    5. How to jump-start your IoT integration journey

What is IoT?

It is the internet connectivity of small devices such as sensors, actuators, smart appliances so they could share and gather information to and from each other.

Some of you might be asking, what’s the big fuss? We already have the internet, and most of you in the power sector already implemented SCADA and PLC before. So how is it any different?

IoT addresses a lot of shortcomings of SCADA and PLC. That is a new level of connectivity between devices. The main difference is when any components within the environment are interconnected, it can act on its own without any human intervention. With all the information it has, it provides rich insight into its entire surrounding environment and instantaneously reacts efficiently by making automatic changes and actions. It can also improve performance and even predict failures, just like Tesla cars or autonomous machines that are in manufacturing plants.

To fully appreciate all the benefits of the internet of things in the energy sector, all of its major components, such as generators, grids, distributors, and consumers, should go smart soon as smart grids cannot function without ‘smart’ consumers.

What is an IoT Platform?

An IoT platform is an integrated service that connects devices and generates its ecosystem online. In the simplest terms, an IoT platform is a group of systems and services where you could operate your entire IoT landscape.

IoT Platform Types, Services, and Offerings

End-to-end IoT Platform

Some companies offer end-to-end solutions. They sell everything – hardware, actual sensors, services, software, and connectivity, plus the device management tools. It is a one-stop-shop, but this tends to be a lot more expensive. You could end up buying features or components that might not apply to your current needs, and yet, you are paying for the entire cost. One of the downsides is that it could become a case of classic single vendor lock-in that most companies steer away from.

Connectivity Management Platform

It is an inherent part of the IoT technology stack. It is the one responsible for connecting and converting data to and from devices and IoT core platforms, regardless if it is on-premise or on-cloud. The primary use case for this type of platform is when you partner with a mobile network to handle communications for security purposes. It means that the data would have to travel through a cellular network rather than flow directly to the internet. But of course, it also introduces new challenges and complexities. So you need to establish your business requirements first.

IoT Edge Platform

We see here the edge devices and the communication gateway. In this type of platform, they provide actual devices and software or tools to manage and configure the devices remotely. With this type, you don’t necessarily need to connect to the cloud as these edge devices are, in a way, functioning already as mini computing and storage machines.

IoT Cloud Platform

It is the most common computing strategy. In this setup, all information from IoT devices could be uploaded to the cloud and a centralized repository. It then runs insights with the aid of full-blown machine learning and artificial intelligence tools. Such simplifies the management of device configurations, managing, and monitoring. Whatever you deploy into the cloud, you could also have on-premise. Your cloud could also push data to the on-premise platform or vice versa.

Edge Computing

An essential part when deciding what type of platform you will choose is whether you would go with edge computing, cloud computing, or hybrid.

Edge computing is bringing the processing at the edge of the local network, not the office LAN, but the network within the autonomous component, for example, a smart car and smart parking garage. These edge devices could send and receive from various IoT devices and communicate directly with other edge devices. 

A case in point is the autonomous Tesla cars that can park itself in an IoT enabled parking garage. The cars going in and out of the garage did not wait for details from the cloud. Instead, they talk to each other directly. Such solved a crucial problem associated with the centralization of data in the cloud. In this model, all processing is distributed, and decision making is down locally by bringing cloud computing capabilities and localizing the data to these devices. It can reduce latency, process data faster, and provide instantaneous insights, as compared to sending everything first to the cloud for analysis.

How Can It Do That?

All the IoT cloud components have been, in a way, miniaturized, streamlined, and deployed in those small edge devices.

Common Components of Edge Devices

Complex Event Processing (CEP) could take data from different IoT devices and act accordingly based on the established patterns. But more often than not, actual CEP models are done on an edge core network. Then, it’s just pushed to the edge devices.

Edge computing devices now support Machine Learning & Artificial Intelligence (ML & AI) locally, but there is always the option of sending the data to the enterprise data warehouse for furth ML & AI processing.

It has the ability to deploy containerized applications as well, either the software package from your solution provider or your own developed applications.

These devices are capable of storage and data management too. These devices now have storage large enough to store all data from different devices, especially in the event of network failure. It is capable of holding data temporarily until the connection gets restored.

When do we consider EDGE over CLOUD?

When your requirements are mission-critical, like, when a machine needs to stop outright to prevent an accident or defective product, sending your data first to the cloud for analysis is not an optimal solution.

Another scenario is when these components are remote, and there is limited or no connectivity to your centralized data center.

Edge computing is still developing, so it has a lot of downsides. Given that the data is localized, security measures might not be as tight as the one securing the cloud security-based systems. It is still a potential privacy and security concern in the edge devices given that the data is stored and already consolidated. That information could also contain sensitive data. Curious people can easily access the device, have the data leaked, or used maliciously.

High availability and failover. Usually, these devices are geographically dispersed. If one edge device goes down, even if you have installed clustered edge devices in one autonomous component, it is still hard to monitor and replace if you have a lot in your IoT plinth.

Scalability. As the number of IoT devices grows, it will be hard to maintain and monitor. Even if you did conduct positive planning, it could only handle so much.

So, choosing between edge and cloud computing depends on your business requirements as well as your business objectives.

Data Integration Platform

The data integration platform is another significant type of platform. Whenever you see an IoT platform presentation, there is always a storage component. It is a combination of several frameworks and tools that will handle the following 

    • Streaming of extracted data from various sources 
    • Routing of data either to your data warehouse or directly to the data consumers’ application
    • Ingestion of data to your data warehouse and OLTP, 
    • Doing a fast filling of data from the database. 

All of these are components of a full-blown enterprise data warehouse or Big Data. What we would like to note is that when you are planning to start your IoT platform, you should first have a data warehouse or Big Data strategy in place.

The goal is to align your IoT storage roadmap with your data warehouse or Big Data roadmap, and not the other way around. Remember that a data warehouse or Big Data setup is also a costly investment and should be treated as a separate project. Having an aligned IoT and data integration strategy could save you a lot of money by maximizing the use of your entire infrastructure.

There are a lot of components that IoT and data warehouse share – streaming, ingestion, and routing. All of these can have its cluster but can be utilized by both your IoT and data platform. With this setup, you can save cost in your infrastructure and reduce the skillset that you need from your IT team.

IoT and Big Data share the same challenges – volume/size of the data coming in, velocity/speed, variety, and veracity/accuracy. When deciding on your IoT storage platform, you have to make sure that it could address all of these challenges.

Here are the few options that you could opt to adopt for your IoT and data warehouse platform.

So, when you have to decide as to which computing strategy or data platform to employ, you need to consider a lot of factors. Don’t just jump on the bandwagon.

Speaker: Mr. Steven Siahetiong | Exist Technical Architect

IoT Components

  1.     Things pertain to sensors, actuators, or any device capable of sending data to the cloud.
  2.     For these devices to send data, it will need to use networking and messaging protocols for the data to be transferred to its destination. Most use wireless access for network connectivity. During the past few years, there have been substantial developments in wireless connectivity protocols. Some of the examples are Bluetooth, Low Power WAN, ZigBee, 6LoWpan, and Thread.
    • Bluetooth is a global 2.4 GHz personal area network for short-range communication. 
    • LPWAN is a type of wireless telecommunication wide area network designed to allow long-range communications at a low bit rate. There are several competing standards and vendors in the LPWAN space. The most prominent of which include Laura and SigFox.
    • ZigBee is a 2.4 GHz mesh local area network lan protocol. It was originally designed for building automation and control, so things like wireless thermostats and lighting systems often use ZigBee.
    • 6LoWpan uses a lightweight IP based communication to travel over lower data rate networks. It is an open IoT network protocol like ZigBee and is primarily used for home and building automation.
    • Thread is an open standard built on IPV6 and 6LoWpan protocols. You could think of it as Google’s version of ZigBee.

For the messaging layer, the most popular are HTTP, MQTT, and CoAP.

    • The CoAp protocol is a client server-based protocol that allows pockets to be shared between client nodes, which are commanded by the CoAP server.
    • The MQTT protocol is communication-based, which is based on the publish/subscribe methodology in which clients receive information through a broker only to the subscribed topic.
  1.     The IoT solution needs to have a platform. An IoT platform combines several IoT functions in one. It can collect and receive your data, convert data between protocols, and store and analyze data. They are available as cloud-based and standalone platforms and are available for many companies, both large and small.

The cloud will have an important role to play in IoT as it will enable companies to create networks, store data, and automate processes without having to build the infrastructure themselves. Such will let IoT services to be developed much quicker and at a lower cost than using traditional in-house systems and services.

As seen on the graph, HTTP involves the largest bandwidth and latency than any other protocols, while CoAP has the least bandwidth and latency.

MQTT offers the highest level of quality of services, with the least interoperability among the four. On the other hand, HTTP was designed for the greatest interoperability on the web and did not include reliability as a core feature.

There are many IoT platforms in the market, and the functionality of these platforms varies enormously. Although all IoT platforms will have dashboards to display data, some platforms are essentially dashboards and are only capable of displaying data from devices. You will often find the terms dashboard and platform used interchangeably.

An IoT dashboard can be considered as a basic IoT platform. A dashboard can usually display data and control devices. However, an IoT platform can usually collect data from various sources, store data, control devices, display data, run tests, deploy device updates, and manage device inventory.

Concepts

Events, insights, and actions are functional concepts that exist across the devices, platforms, and applications of an IoT solution. To further explain, consider an application that monitors the cooling system, temperature for food storage, and calls emergency maintenance services if the temperature becomes dangerously low or high.

The following processes occur in this example:

    • The devices send temperature samples from the primary cooling system to the IoT gateway via the device to cloud events every 30 seconds.
    • These events can generate insights. The IoT platform can evaluate events for any immediate contextual insights, such as temperatures at malfunctioning levels.
    • The generated insights can trigger actions. If the temperature is at a malfunctioning level, the platform can send a command to the backup system to start while the maintenance is en route to the location.

Events represent the device-to-cloud communication in an IoT solution, and maybe notifications, acknowledgments, or telemetry data.

Insights are interpretations of events. It may derive from events directly as contextual insights or transform or stored event data by application event processing for real-time or aggregated insights.

Actions are activities undertaken either programmatically or manually as a device, service, or analog actions.

IoT Customer Scenarios

How can you design an IoT solution to satisfy customer requirements?

    • Manufacturing safety systems must respond to operational data with ultra-low latency and control.
    • Mission-critical systems such as remote mining equipment, connected vessels, or offshore drilling need to analyze and react to data even with limited connectivity to the cloud.
    • The volume of data produced by jet engines or connected cars can be so large that data must be filtered or pre-processed before sending it to the cloud.
    • Regulatory compliance may require some data to be locally anonymized or aggregated before being sent to the cloud.

IoT Edge

IoT devices can connect to the IoT platform directly or through IoT Edge gateways that implement intelligent capabilities. With an IoT edge, you can analyze censored data in near real-time and issue commands when you detect anomalies to stop a machine or trigger alerts. Your streaming logic runs independently off the network connectivity, and you can choose what to send to the cloud for processing or storage. You can also filter or aggregate the data that needs to be sent to the cloud.

The benefits of using edge gateway, particularly in an IoT application, is that moving data processing functions from the cloud to the edge helps ensure accuracy and reliability. Transmitting data to and from the cloud takes time. Even milliseconds are too long for many mission-critical decisions and processes that occur in industrial operations. In the time it takes to send data to and get a response to the cloud, the data could simply become obsolete, resulting in missed opportunities for action, damaged components/products, or risks to the safety of the equipment/personnel.

Edge devices can also send data or commands directly to other devices for immediate action. Edge computing via edge gateway also makes it possible to keep sensitive data on-site to ensure its security. Some devices or systems generate so much data that the bandwidth and resources needed for the cloud to handle this data are too costly. The edge gateway can determine which data can be sent to the cloud and transmit it in the most efficient and usable form.

Industrial Edge LORA 1 Gateway

    • Extremely flexible and powerful
    • Allows to run Lola 1 application and network server locally to setup on-premise private network
    • Aimed to help the transition from legacy to connected automation with its ability of interfacing existing devices.

Types of Edge Gateways

IoT Edge devices can act as communication enablers, local device control systems, and data processors for the IoT cloud platform. IoT Edge devices can run cloud workflows on-premises and can communicate with devices even in offline scenarios.

Cloud gateways can do protocol and identity translation to and from the IoT cloud platform and can execute additional logic on behalf of devices.

In a transparent gateway pattern, the gateway simply passes communications between the devices and the IoT cloud platform.

A protocol translation gateway is also known as an opaque gateway, in contrast with the transparent gateway pattern. In this pattern, devices that do not support MQTT, AMQP, or HTTP can use a gateway device to send data to IoT cloud platforms on their behalf. All information looks like it is coming from one device, the gateway. 

Exist, as a technological solution provider, designed a generic IoT and data integration reference architecture for our customers, especially for the power industry.

The Data Integration layer handles the ingestion of the various data sources. It supports all types of connectors.

IoT Gateway acts as a gateway to connect your devices and accepts data streams using MQTT, CoAP, and HTTP messaging protocols. Data can be ingested as a batch or a real-time stream. With batch processing, data is collected in batches and then fed into an analytics system. A batch is a group of data points collected within a given period. Unlike stream processing, it does not immediately feed data into an analytics system, so results are not available in real-time.

With stream processing, data is fed into an analytics system piece by piece as soon as it is generated. Instead of processing a batch of data overtime, stream processing feeds each data point or a micro-batch directly into an analytics platform. This allows teams to produce key insights in near real-time.

Streaming platforms allow data transformation persistence and allow interactive queries on the streaming application. Kafka is the most popular choice among streaming platforms. 

ETL is the extract, transform, and load process for the data. While the ELT is the extract, load, and transform process. In ETL, data moves from the data source to staging and into the data warehouse. ELT lets the data destination to the transformation, eliminating the need for data staging.

Optionally, you can add a dataflow tool such as NiFi to provide a web UI, which design or control your data pipeline in a graphical representation. NiFi can deal with a great variety of data sources and formats.

For Stream Analytics, the popular choice is Apache Spark Streaming. It is a scalable and fault-tolerant stream processing system that allows data engineers and data scientists to process real-time data from various sources.

Data can be processed using complex algorithms, expressed with high-level functions. Finally, processed data can be pushed out to file systems, databases, and live dashboards. You can apply Spark’s machine learning and graph processing algorithms to data streams.

For the ETL jobs, you can apply data quality steps before loading it to a data warehouse. While for ELT, data is simply dumped into the data lake, and transformations happen on an as-needed basis, and only the data that needs to be analyzed at that time are transformed.

Hot path

    • analyzes data in near-real-time, as it arrives
    • events must be processed with very low latency
    • typically implemented using a stream processing engine
    • the output may trigger an alert or be written in a structured format that can be queried using analytical tools

Warm path

    • holds data that must be available immediately from the device for reporting and visualization

Cold path

    • performs batch processing at longer intervals (hourly or daily)
    • typically operates over large volumes of data, but the results don’t need to be as timely as the hot path.
    • captured and then fed into a batch process

Machine Learning allows predictive algorithms to be executed over historical telemetry data, enabling scenarios such as predictive maintenance. It requires training by telling a machine learning model what it’s trying to predict, similar to how a human child learns.

Speaker: Mr. Jejomar Dimayuga | Exist Technical Architect

Industrial IoT Stack

Sensors

    • Actual things that are located in your physical environment
    • Smart meters or actuators

Edge Devices

    • Can act as the concentrator/aggregator for all of your devices
    • Allow different protocols depending on the support of your devices

Gateway

    • Connectivity to your edge devices

Cloud (or on-premise infrastructure)

    • Where your applications run
    • Offers high availability and high scalability solutions

Insights

    • Where analytics are produced that can be based on historical events, failures, rules, or condition that can communicate back to your device
    • Where algorithms are performed after ingesting and storing the data

Consumption

    • How your data is consumed
    • How can it be displayed via the web, iOS, or Android devices

Security and encryption across the components are must-have. It is a vital requirement for all systems, especially in the IoT solution for energy. Security components such as SSL or HTTPS, LDAP Integration, Single Sign-On, Access Token are already in place.

We accommodate Access Token so we can ensure that your devices are secured in connection to your edge devices and IoT gateway.

We also offer a Single Sign-On authentication to your existing authentication features via the Oauth2.0 protocol.

We can also integrate your authentication via LDAP. We can connect your authentication and your road-based access control to an active directory in your organization if you have one.

To secure your communication layer, we can incorporate SSL or HTTPS with certificates from the latest update of the TLS version.

Another important aspect of any system nowadays is scalability and high availability. In this solution, we can incorporate Docker containerization via the Kubernetes platform. In this way, we can deploy your application or your IoT gateway into multiple replicas to ensure that it is highly available. We can:

    • Reach the maximum daily quota of your required messages
    • Reach the quota of connected devices 
    • Increase the ingestion throughput
    • Increase the processing throughput
    • Set up a DR environment in place to ensure that your operations will be continuous

All of these are in place already so that we can ensure that you will be able to perform your analytics and your business needs. On top of that, we can also integrate into your existing systems, may it be your billing system, billing, or offer system, CRSS, or else.

Speaker: Mr. Chris Silerio | VP for Operations

Things to Consider on Choosing an IoT Platform

    • The capability of your team
    • Geographic coverage
    • Timeline and budget
    • Your IoT and EDW roadmap
    • Your business billing model
    • Scalability of the platform
    • Method of connectivity
    • Integration flexibility
    • Device management
    • Cloud VS. on-premise

Developing your platform in-house is fine as long as you are not pressed by time. Otherwise, it might be better to start with a simple platform and develop on top of it.

Q&A

Q1: With the vast array of open standards for all the layers of IoT, is there a considered safe or conservative stop that anyone can use as a basis or a starter that is easy to pivot towards specific solutions as requirements get clearer? 

A: As shown on the Industrial IoT Stack diagram, we could have the base layer of the devices, followed by the edge devices if you want to go with edge computing. After that, we can set up an IoT gateway on cloud infrastructure.

It also depends on the network or protocol that the sensors, actuators, or your device are supporting. What matters most is that your platform can connect with these various open standards and not limit your entire solution. If there is a new protocol that comes in, your platform should be flexible enough to support these open standards.

 

Q2: What are actuators?

A: Actuators are mostly used in factories. These are devices that are connected to types of machinery. It typically sends telemetry data, like temperature, weight, the roughness of a substance.

 

Q3: What is the experience of Exist in developing IoT systems for energy efficiency applications?

A: We have a couple of projects on the integration of IoT devices, and it is still growing. Our forte is software development integration. But our experiences with these projects gave us enough exposure to the entire ecosystem.

 

Q4: Have you done any IoT applications for use in the Philippine market?

A: We are developing some right now for companies who are just starting their IoT projects. These companies are doing it in phases. We are currently in the first phase. (Company names cannot be disclosed yet.)

 

Q5: Given the poor internet connection in the Philippines, would IoT applications that HTTP use be a challenge?

A: High likely, cloud computing would be a challenge because it will always depend on internet connectivity. In scenarios that it is in remote areas, it would be better to go with edge computing provisioning. That is one of the advantages of an edge over cloud computing. 

Aside from internet connectivity, some providers also offer alternative communication layers, such as cellular networks.

 

Q6: Do you provide a complete solution, like sourcing sensors or edge devices, depending on the requirements?

A: No. There are only a few companies here in the Philippines that offer end-to-end solutions. We are not a hardware company. We are agnostic of software and hardware. We do more on the development of all these layers/components from the IoT devices up to the data integration. But we are partnered with companies that we may utilize to provide an almost end-to-end solution.

 

Q7: For an organization that has mostly in SCADA plus TLC equipment, what would a migration plan look like when moving to a more modern platform that allows a more open choice of technology stack?

A: IoT is not intended to replace SCADA and TLC. It is a different level/layer. Information from SCADA and TLC may be integrated into your IoT platform. We do not have to migrate as we can support SCADA and TLC.

 

Q8: With regards to data storage, would you still recommend Hadoop, or are there better alternatives?

A: We recommend that you go with the storage that can handle multiple parallel processing and the other factors that would support volume processing of data, and speed from streaming data, flowing data from different data sources to your storage devices. There are a lot of databases that support multiple parallel processing, like Greenplum, Oracle, Teradata, and Hadoop.

 

Q9: If I were to engage for an IoT application, what would be the start?

A: It might be better if you could consult first with a third-party company that already has the experience or an offering of IoT solutions in the market. It is quite hard to say outright what the specific software that we are going to use in your IoT platform. We need to undergo some discussions within your organization before we could provide a straightforward platform that you could use. We could always start simple. But regardless of how simple it is, we still need to know what are the factors that we should consider for your organization.

 

Q10: Can you offer an outsource service to support an IoT application?

A: We could somehow handle it, given that we have partners. You may also directly talk to hardware vendors, especially about edge computing devices. We can also give you a list of vendors that you could contact for each IoT component

Database. Java. Java Philippines. PostgrEX

Introducing PostgrEX: How to Fulfill Your Database SLAs in 2020 Without Having to Sell a Kidney

Introducing PostgrEX: How to Fulfill Your Database SLAs in 2020 Without Having to Sell a Kidney 768 487 Exist Software Labs

In a past blog post, I gave the definition of software as being enterprise-grade in the following manner:

A piece of software is enterprise-grade when it caters to the needs of not a single individual, nor a select group of individuals, but the whole organization.

When applied to database management systems, an enterprise database is an implementation of database software that serves the organization by managing their humongous collection of data. It must be robust enough to handle queries from hundreds to tens of thousands of users at a time. It must also have a host of features that are geared towards improving the productivity and efficiency of the organization, such as multi-processing, parallel queries, and clustering, to name a few.

To tease it out a little bit further, I would like to propose that a database implementation is “enterprise” when it possesses the following attributes:

1.    A database engine that has proven itself in a multitude of business applications globally in a span of decades

2.     Able to meet strict SLAs (at least 5 nines) through high availability and failover mechanisms

3.    Monitoring

4.    Backup and Recovery Management

5.    Connection Pooling

Traditionally, enterprise database implementations have been costly investments and organizations have been willing to pay the price given the criticality of data to any business endeavor. But given the current economic climate brought on by the COVID-19 pandemic, along with the perennial need for businesses to streamline costs in order to divert savings into the core business, many are asking: Is there a better, more cost-efficient way of implementing a database solution without sacrificing enterprise-ness?

The answer is most certainly! Let me introduce you to PostgrEX.

What is PostgrEX?

What is PostgrEX?

PostgrEX is shorthand for Postgres EXIST Enterprise Xpertise.

It is an enterprise-grade database platform built on top of a purely open-source technology stack and is part of EXIST Software Labs Inc.’s Data Solutions.

What are the components of PostgrEX?

1.    Scoping and sizing of DB hardware

We will recommend the hardware specifications (memory, CPU, storage, networking, etc.) that will be optimal for your business requirements based on the current and projected data growth, the total number of users, total concurrent users, largest table size, largest query size, etc.

2.     Installation

We will install the database system, along with the high availability/failover, monitoring, backup/recovery, and connection pooling components.

3.    Optimization

We will optimize the database configuration settings for the best possible performance given the hardware available.

4.    High Availability/Failover/Disaster Recovery

We will set up replication between the Postgres database servers (streaming replication, WAL log-shipping, or a combination of both) in the Main site and we can also set up replication to a DR site.

We will also set up and configure Patroni, etcd, and HAProxy as part of the failover mechanism of the system.

5.    Monitoring

We will install, set up, and configure pgCluu as the default DB cluster monitoring tool.

6.    Backup and Recovery

We will install, set up, and configure Barman as the default DB backup and recovery management tool.

7.    Connection Pooling

We will install, set up, and configure pgBouncer as the default DB connection pooling tool.

8.    Query Optimization

We can also provide query optimization services to your Developers in order to ensure tip-top application performance.

9.    Migration to Postgres

We can migrate your existing SQL Server, MySQL, and Oracle databases to Postgres CE.

What are the technologies used by PostgrEX?

1.    Database

Postgres, or PostgreSQL, is arguably the best open-source object-relational database management system available today. It was DB-Engine’s “DB-of-the-Year” for 2 years straight (2017 and 2018), and has proven itself in mission-critical applications across all industry verticals.

See: Why use PostgreSQL for your Business?

2.    High Availability and Failover

Patroni – an open-source Python application that handles Postgres configuration and is ideal for HA applications. See Patroni documentation.

etcd – a fault-tolerant, distributed key-value store that is used to store the state of the Postgres cluster. See etcd documentation.

HAProxy – provides a single endpoint to which you can connect the application. It forwards the connection to whichever node is currently the master. It does this using a REST endpoint provided by Patroni. Patroni ensures that, at any given time, only the master Postgres node will appear as online, forcing HAProxy to connect to the correct node. See HAProxy documentation.

3.     Monitoring

pgCluu – a lightweight, open-source Postgres monitoring and auditing tool. See pgCluu documentation.

4.    Backup and Recovery

Barman – an open-source backup and recovery management tool. See Barman documentation.

5.    Connection Pooling

pgBouncer – a lightweight, open-source connection pooler for Postgres. See pgBouncer documentation.

Moving Forward with PostgrEX

Is your organization ready to face the challenges of an uncertain future? Having enough money in the bank is certainly a top priority and doing away with unnecessary and exorbitantly-priced database license costs is one way of doing this.

With PostgrEX, your business applications can still enjoy industry-recognized, top-level, enterprise database excellence through the use of expertly-configured, purely open-source technologies. This means you get to keep your kidney to live and fight another day—and many other days!

Contact us for more information.

Download our datasheet now!