Transmetrics 21.03.2018 27 min read

Big Data And Big Roadblocks: How the Logistics Industry Can Overcome Its Big Data Challenges

This blog has been written by Transmetrics. Please visit their website for more information or find out more about them below the post. We would like to thank them for their valuable insights on the topic. 

The modern transportation and logistics (T&L) industry is praised for efficiently moving goods across the world, yet the only thing that doesn’t seem to be moving quite as fast is the industry itself. As this sector has been slow to adopt and implement emerging technologies, e-commerce giants like Amazon and Alibaba have stealthily developed their own intricate logistics networks, both threatening traditional logistics companies, and highlighting the latent power of supply chain data in the process. 

However, while they say you can’t teach an old dog new tricks, T&L seems to be learning how to roll over on its legacy mentality, in attempt to avoid playing dead for real. It has become increasingly clear that, in order to compete with this new breed of e-commerce/logistics companies, the industry’s only way forward is to leverage data for actionable business insights. In fact, one study found that 98 percent of third-party logistics companies (3PLs) and 93 percent of shippers believed data-driven decision-making was essential to supply chain activities.

Accordingly, IoT adoption in T&L has been growing fast, and pumping out data even faster. At the same time, artificial intelligence and machine learning have made it easier than ever to turn that data into meaningful insights. But despite the emerging volume of data (and the opportunities it carries), a number of challenges are preventing the industry from effectively leveraging the data at its disposal. Here are some of those challenges, and more importantly, how to overcome them.

Legacy mentalities

Though the industry has started to jump on the digital bandwagon with its adoption of IoT and AI technologies, it still has a long way to go before it can make the most of its data. T&L faces a serious challenge in regard to educating industry operators and bringing them on board with the latest technology. The truth is, most people in the industry still adhere to an old way of thinking and doing. A prime example of this is the fact that Excel still remains one of the primary tools in the industry, despite its inherent limitations. Dependence upon this software is so extreme, in fact, that it is actually the industry’s worst enemy.

In order to change these legacy mentalities, there is a need to educate the industry. As more and more impressive technological solutions hit the market, their creators will need to get involved at industry conferences to share their knowledge and success stories, in addition to creating informational posts such as this.

Fortunately, the seeds of change are already being sown. In recent years, a new position in the industry has emerged. “Chief Digital Officers” are slowly beginning to get the industry on track in regard to leveraging its data. As this trend continues, the industry would also benefit from bringing young, technologically advanced employees on board, as well as people from other industries in order to spark new ways of thinking.

Fragmented data between industry players

Assuming T&L can modernize its mentality, other significant challenges still remain before the industry can effectively leverage its data. One of the biggest challenges is the fragmentation of data across the supply chain. There are many players in the logistics industry, including suppliers, shipping companies, warehouses, and last mile delivery services among others, and each of the players has visibility and records data over only a portion of the process.

This means that, unlike Amazon, logistics companies have no single visibility/single version of truth over the entire supply chain. For data to truly be effective -- and for logistics companies to provide the best possible end-to-end delivery service -- the existing gaps must be closed. If they aren’t, logistics companies won’t be able to compete with companies like Amazon, and will suffer from huge efficiency losses that affect both customer satisfaction and their bottom lines.

In fact, just recently, Maersk, the world’s biggest container shipping line, revealed its concerns about e-commerce companies like Amazon and Alibaba encroaching on its business. “If we don’t do our job well, then there’s no doubt that big, strong companies like Amazon will look into whether they can do better themselves,” said Maersk CEO Soren Skou in an interview with Bloomberg.

In order to overcome these issues, logistics companies should try to negotiate with their suppliers to establish cooperation and get them to share their data. In many cases, suppliers know what they are going to ship, but most of them don’t care or bother to share the data with the transporters. Convincing suppliers to share their data might require logistics companies to offer incentives such as discounts, but the extra visibility gained would be well worth the cost. Market researchers, as well as AI and machine learning, could also be used to better understand industry behavior and gain more visibility over the entire supply chain.

Inconsistent and low-quality data

Additionally, due to the number of players involved in transporting goods from the supplier to the consumer, data is often inconsistent and recorded in an unorganized manner. Different players record their data in different systems, and sometimes even in different units of measurement (weight, dimensions, metric/imperial, etc). As a result, it is very difficult to reconcile the data and analyze it to glean powerful insights.

But the industry’s data is not only inconsistent; it is also low quality and inaccurate. Take this example of Target, where barcodes on many items did not match what was in the computer system, causing goods to come into the warehouses much faster than they were going out. The fault could have been from Target’s vendors, buyers or warehouse computer system, but regardless, the result was the same: the company’s supply chain clogged up, and it lost nearly a billion dollars as a result.

Extremely low-quality data has made it difficult for logistics companies to analyze trends and make informed business decisions. As a result, business owners often must rely on the subjective judgment of middle management to determine the current state of the business and its operations, instead of reviewing concrete data to reach clear, educated conclusions.

To tackle this problem head on, the first step is for logistics companies to develop and adhere to certain standards for recording data, so as to avoid using different types of measurement, and facilitate data analysis. The Logistics Interoperability Model (LIM) was released in 2007 to do just that, but many companies fail to follow it, instead recording data however they see fit. Once companies in the industry begin to recognize the network benefits of a standardized model, the next step is to spend time enriching their data to prepare it for analysis.

Additionally, companies can share their platforms and data with each other to ensure accuracy. However, most people are afraid that this type of data sharing will make them lose ground to their competitors. Therefore, the ideal solution is to agree on specific rules of data sharing in a way similar to the Physical Internet, in which academia, industry and governments across the world share their data and work together to transport goods in standard-sized, modular containers as seamlessly as the internet moves digital information.

This model is similar to the previously described LIM, however, it incorporates more parties, and has the potential to impact the industry in an even bigger way. If such a standard eventually emerges due to a legislative initiative, the industry will have to start following it to avoid facing the same problems in the future. But again, this will only work if the data is clean.

A number of software solutions are on the market to help to overcome big data challenges, but not all of them specialize in logistics. Transmetrics is able to take logistics data, clean it and enrich it to help optimize planning processes within cargo companies by using modern technologies such as Artificial Intelligence and predictive analytics. Additionally, YourEDI can detect if certain fields are missing from the data, and automatically forward back to the sender and ask for modification. Together, solutions such as this can help logistics companies take the first step toward using their data to stay competitive against e-commerce giants like Amazon.

Lack of data experts

Even with a large amount of clean data, as well as tools that can identify patterns and trends that humans wouldn’t have been able to, it is difficult to know exactly what data to analyze. Moreover, it’s easy to make the wrong conclusions. Often, people who are not experts might mistake a correlation effect as a causation which can ultimately lead to the wrong business decisions. “When you have too much data you can find random correlations that have no real causal link, which could lead a company down the wrong path,” explains Paul Brody, EY Americas Strategy Leader, Technology Sector, in a report about the digital supply chain.

Therefore, it is crucial that the industry begin employing experts in data analysis. Given the proliferation of data in the industry, non-technical employees will not be equipped to adequately manage the new amount of information. Looking to hire technical employees skilled in data analysis, or re-training current employees, are both good options for companies in the industry moving forward. These employees will be able to advise business owners about high-importance business decisions – instead of having to rely solely on the opinions of middle management. However, attracting technical employees to the industry is no easy task. For this reason, logistics companies should look to implement existing software solutions for logistics that can help guide their decision-making processes.

Security vulnerabilities

As with any data-heavy industry, data security is a huge concern. And now that the number of connected devices within logistics is growing rapidly, the risk facing the industry is greater than ever. In a post for Forbes, data expert Bernard Marr writes, “As Big Data increases in size and the web of connected devices explodes it exposes more of our data to potential security breaches. Many organizations already struggled with data security even before the complexities added by Big Data, so many of them are drowning to keep up.”

T&L is no exception. Companies in the industry possess a large amount of personal information, including consumers’ home addresses and credit card numbers among plenty other sensitive information. While keeping this information safe was a challenge before all of the connected devices, it is even more difficult now. An EY report, in fact, labeled compliance risk as a major challenge in the industry due to the fact that, on average, 54 percent of the data collected by companies was “dark data,” with its contents unknown.

Blockchain has a lot to offer in addressing these data security issues. If data were to be recorded on the blockchain, permissions could be strictly regulated whereby it would require multiple authorizations from across the network before the data could be accessed. Blockchain technology has already shown promise for the healthcare industry which manages highly sensitive patient records, and it could do the same for logistics. However, in order to work effectively, data must first be cleaned, as it is impossible to change once it is recorded on the blockchain.

With IoT, AI and blockchain, the T&L industry is set to reap huge benefits. In fact, one report predicts that IoT will bring about a 15 percent increase in productivity for delivery and supply chain performance. Another report forecasts AI to drive GDP gains of $15.7 trillion globally through productivity and personalisation improvements, naming T&L as one of the sectors with the biggest potential for impact.

While the industry is just starting to get on board with the move toward digitization, it also faces a number of fundamental problems that threaten the technologies’ potential for success. In order to overcome these challenges, and effectively leverage the data they have at hand, players across the industry will need to work together to effect major change. As this change comes, companies like Transmetrics and YourEDI help push the industry to maximize its potential by harnessing the power of big, clean data.

About Transmetrics


Transmetrics is a Big Data, predictive analytics solution which was created exclusively for the cargo transport industry. By applying AI and machine learning, Transmetrics products is solving the existing inefficiencies of the industry concerning capacity utilization. The product implementation consists of multiple steps: data cleansing, advanced forecasting, predictive optimization, and execution controlling. Transmetrics' products help companies make better, data-driven decisions and continuously improve their operational performance. Transmetrics works with the world's leading logistics companies, including global Fortune 500 companies.In 2017 the company launched its blog, which discusses transport, logistics, and innovation globally and in Europe.

Do you want to learn more about the topic? Download our Transportation Logistics eBook:

Download Now



Transmetrics is a provider of SaaS (Software-as-a-Service) solution for the transport and logistics industry

Speak to us About your Integration Needs!

If you have integration-related challenges or questions about Youredi's integration solutions, please book a 30 min time slot to discuss them in-depth with our team.
Book a Call



The latest News and Insights from the Supply Chain and Logistics Industry.