Barbara Babati 21.05.2019 17 min read

Do We Need Data Standards or Data Transformation Technology?

Recently, standardization has been a hot topic in the container shipping industry. Organizations, like UN/CEFACT and the Digital Container Shipping Association (DCSA), have been working on introducing new standards that stakeholders of the maritime industry could utilize.

Data standards have been around for decades. Their original purpose was to simplify business-to-business communication. Nevertheless, the new sub-standards of data standards emerged. Instead of having a single standard streamlining the data exchange between the stakeholders, it just got more complicated as sub-standards and in-house data formats manifested.

 

Different systems and applications use different data standards and data formats. Changing the actual data standard of a system is not easy. Therefore, the adoption of new standards could be a challenge. The result would be that we’ll need to deal with new standards besides the previous ones.

 

At the same time, the technology for data harmonization is mature and imperative in trading partner data exchange.

 

So, in the end, the question is: do we really need a new standard? Could we just use technology to transform the data standard into one’s preferred format?

 

This article will discover the benefits and challenges of data standards. Perhaps standards that were developed for EDI (electronic data interchange) are the most known – however, even the de facto EDI standards have variations which make it more challenging to work around the diversity of standards. Another challenge regarding data standard is that APIs (application programming interfaces) are becoming essential in B2B data sharing. This means that working around legacy systems (and the standards they use such as EDIFACT, X12, or proprietary formats) and formats that APIs consume (e.g., JSON and XML) is something that new standards cannot help to overcome.

 

Benefits of Data Standards

 

In this article, we are disputing the need for developing new data standards. Nevertheless, as a technology vendor, we clearly see the benefits of using the available standards and why would container shipping stakeholders believe in new standards and their positive impact on the industry as a whole.

 

As mentioned above, standards were created to simplify trading partner communication. In an ideal world, everyone would be using the same data standard with the exact same structure so there wouldn’t be a need for data mapping and translation.

 

Data standards are also well-documented as they were created to be widely used, thus working with these standards is typically easier than working with poorly documented in-house data formats. A data standard needs to have clear elements, definitions, and formatting rules.

 

Using standards for data exchange can improve the data quality and compatibility, as well as the data collection modes and the reusability of the information. At the same time, standards also reduce redundancy, time and effort spent on reconciliation, the number of iterations, manual interventions, exception flows, corrections, and last, but not least the cost and complexity of the information exchange.

 

Challenges of Data Standards

 

While standards have many benefits, the development of a new standard also comes with a lot of challenges.

 

First, the organization that wants to develop a standard needs to have a clear strategy, as standardization requires clear focus. The scope and specifications of the standardization project should be realistic. To execute the standardization strategy, strong leadership is a must.

 

Once the strategy and the scope are clearly defined, the development phase of the new standard will be even more difficult. It will require a vast amount of resources, both in terms of labor and investment. The development phase can last for years. If you look at the timeline for adoption and roll-out for the PEPPOL standard, you will see how long the process can be. We just had the 10th anniversary of PEPPOL, and still, we are nowhere near to even having a Pan European standard in place in practice for one message type (i.e., invoice).

 

This year, after years and years of lobbying, we have also seen a few other countries getting on the bandwagon, namely Australia and New Zealand. The rest of the world could not care less about the adoption of PEPPOL, as they already have their own standards (like pretty much all the South-American countries).

PEPPOL standardization timeline

 

PEPPOL standardization second phase 

 

This also means that the last phase, the adoption, has to be planned and driven carefully by the organization that developed the new standard. Ensuring that all container shipping stakeholders would adopt the new data standard is an enormous challenge. These stakeholders may fear that they will lose control over their own data specifications and that they would need to implement changes that they would not necessarily need. An even bigger issue is that legacy systems cannot adopt new standards, so no matter the newly developed standards, there’ll still need for data mapping and harmonization.

 

Levels of Data Standards

 

While we generally we are just talking about data standards, standards also have different levels by nature. The reason for that is that data is dynamic, which means that it does not have “one shape or form”, but instead, it keeps evolving and changing. Besides this, also the data sources vary so much that standardization becomes a challenge, as some data sources already deal with a specific standard or format that cannot be changed or it would be too risky.

 

We’ve defined three different categories for data standard:

 

Set data standard: Set data standards provide detailed rules on how to implement data policies. Such data standards are, for example, RosettaNet or in-house data standards.

 

Forced data standard: Consortiums and/or single entities may set data standards, and they would force all members and parties to use the data set strictly when they communicate with the consortium. An excellent example of such a data standard is PEPPOL.

 

De facto data standard: A de facto data standard is a custom or convention that has achieved a dominant position by public acceptance or market forces (for example, by the early entrance to the market). The best example for de facto data standard is EDIFACT and ANSI ASC X12. It’s sporadic that a new data standard would end up in this category.

 

EDI “Standard”

 

The variety of EDI standards is proof of how difficult it is to develop a standard that would be universally used.

 

EDI (as a technology) was aimed to simplify business-to-business communication. But instead of a single standard, quite a few ones evolved and it presents a challenge for companies. In only rare occasions, multiple parties use the same data standard with the exact same specifications. This means that data mapping and data translation cannot be avoided during business-to-business messaging.

 

EDI data standards 

But does the existence of multiple data standards really an issue? In the end, it shouldn’t be as long as we have a “translator” that can ensure that in the end, we’ll be able to understand each other.

 

Data Standards vs. Data Translation/Harmonization Technology

 

The real question of this blog is whether we need to develop new standards (which will take a long time, it will be expensive, and there is no guarantee for the adoption) or can we just use technology to act as a translator?

 

Standards are exceptional (as you can see all the benefits above), but we will never get to the point when we would only have a single data standard or data format that would be universal, and every system and application would adapt to it.

 

This means that technology will always have a significant role in the harmonization of data formats. Data integration technologies can handle data mapping between any two data formats and standards, regardless of how complex the standard is. Although two endpoints would use the same standards, it wouldn’t necessarily mean that they wouldn’t need data mapping.

 

Data integration technologies for data harmonization are mature, quick, relatively easy to use, and affordable. Depending on your choice, you can either do it yourself or outsource the work to a vendor that provides managed services and has experience of dealing quickly with even challenging data structures.

 

Besides technology can do much more than simply just map and translate the data format. It can take care of other processes as well, such as data validation and enrichment.

 

Bridging the Gap: Hybrid Integration Cannot Happen without Data Harmonization

 

One of the biggest challenge of standardization – and why you may prefer technology over a new standard – is that there is a “gap” between old legacy technologies and cloud-based applications.

 

Old, on-premise systems are prisoners of their own standards. Most likely, these systems won’t adopt new standards, thus it’s essential to have technology at your disposal to work around these. It is especially important when you need hybrid integrations – connecting old systems with new ones. Modern applications often utilize APIs (e.g., REST API), and these endpoints typically use JSON or XML. If you need to transfer data from a legacy system that for example uses EDIFACT or in-house data format to a cloud application via an API, you must have a translation tool in place so that these two endpoints can talk to each other.

 

EDI vs. API

 

As a managed data integration provider, we have recently seen an increase in cases in which we need to tackle integration challenges between EDI and API. In this case, data transformation plays a vital role.

 

EDI systems tend to be legacy ones and often on-premise. While EDIFACT and X12 are the most common data standards used, we’ve usually dealt with other EDI standards, too. Even more common is that systems utilize in-house data standards. Connecting to a REST API is challenging enough, and also, integration architects also need to handle quite complex data mappings.

 

In our case, we’ve developed an EDIFACT and X12 translator to speed up the translation from these formats to JSON and XML.

 

EDI Buyer's Guide Banner

 

Data Standards Will Remain Important, but Data Harmonization Technology Will Be a Must

 

Standards have been essential for decades to ensure that trading partners can communicate with each other. Clearly, data standards won’t disappear, and we are positive that new ones will keep emerging. Nevertheless, to move forward with digitalization in varying industries, standards, and data integration technology will need to work together.

Speak to us About your Integration Needs!

If you have integration-related challenges or questions about Youredi's integration solutions, please book a 30 min time slot to discuss them in-depth with our team.
Book a Call

SUBSCRIBE TO OUR BLOG AND NEWSLETTER

The latest News and Insights from the Supply Chain and Logistics Industry.