Big Data
Big data gives the industry an unprecedented power by bridging both structured and unstructured data and presenting information at the practitioner’s fingertips for quick decision making and insights.
The power of real-time data is unleashed with Open ERP. Odoo Open ERP’s data management capabilities, when
seamlessly integrated, drive and connect enterprise resources with best-of-class analysis platforms.
Data Integration Architecture
Odoo provides an open API and shares freely through a number of standard web service protocols.
For the purposes of this white paper, we will offer several options. To determine the best approach, we would need to understand more about the “Data Lake” and if there is a proposed enterprise solution, such as Hadoop or an Enterprise Services Bus in the works.
"Big data" and "Data Lake" only have meaning to an organization's vision when they solve business problems by enabling data democratization, re-use, exploration, and analytics.
Data Lake
A data lake is a large storage repository that holds a vast amount of raw data in its native format until it is needed. An “enterprise data lake” (EDL) is simply a data lake for enterprise-wide information storage and sharing.
Horton works Hadoop Data Lake
In this case, we are relying on the Hadoop ecosystem to do the heavy lifting, and provide tools that will talk to Odoo’s underlying PostgreSQL database. The Hadoop tool for this is Sqoop, and it uses a JDBC connection to make the connection with Odoo
Any needed Extract, Transform, Load (ETL) activities would be done in Hadoop, using PIG scripting and processing. Hadoop (and the extended ecosystem) are made for this data storage, ETL and integration.
The Hadoop approach may meet the needs that are diagrammed in the “DATA REPOSITORIES” diagram above.
On the Odoo side, we would use the Odoo module “Import_ODBC” to manage record and object changes. This will ensure that Odoo captures “Row/field” level changes, but by going through this module, we can ensure integrity.
Odoo Data Integration Modules
In this case, we are assuming little about the “Data Lake” and its ETL capabilities. We can do the data integration in Odoo with modules that map incoming data to the object models. We can map “like” or “unlike” schemas, using Python and sub mapping.
These tools are part of Odoo’s ecosystem and work exceptionally well. They have provided the foundation of mapping to a number of external systems, such as E-Commerce, Content Management, Enterprise Service Buses, and similar systems.
By using this approach, we are assured of all data being transformed and validated by Odoo itself, ensuring schema cleanliness and row/field level changes.
This approach will work well bi-directionally with data warehousing technologies or other systems. However, maintaining multiple point-to-point integrations can be time-consuming.
Services Architecture
Odoo is usually extended internally via modules, but many of its features and all of its data are also available from the outside for external analysis or integration with various tools. Part of the Model Reference API is easily available over XML-RPC and accessible from a variety of languages.
The most popular services approach is to use XML-RPC.
Please see the documentation here for examples of using this with Python, Ruby, PHP, and Java.
Conclusion
The Power of Data Integration and Open ERP
It’s all about connecting your people by bringing disparate systems and data together. Best practice open-source data integrations will empower your executives to simplify and increase confidence in their decisions. Open Source Integrators will arm your teams to make better, quicker and more accurate decisions to increase the efficiency of the organization and the satisfaction of your stakeholders.