Out of hundreds of questions to be asked during the investment decision process, two seem to be so basic, they can be easily taken for granted: how many units of this asset exist and how much is each unit currently worth?

Digital assets may be one of the greatest innovations since double-entry bookkeeping, but if we apply these two fundamental questions to the majority of them, the answers are murky. The issue of price manipulation in digital asset markets seems to be a widely accepted practice by industry participants and regulators alike. This presents serious problems for the evolution of the digital asset industry. The predominance of bad foundational data has undoubtedly limited institutional participation, negatively affected its acceptance by regulators, and ultimately stifled its growth.

In this post, we present a guide in identifying and quantifying market manipulation, as well as the steps our organization has taken to increase the quality of data for digital assets. The methodology presented here expands upon previous efforts, such as the information made public by Bitwise, and provides an objective and comprehensive way to evaluate primary sources of data.

The Multi-Billion Dollar Hole

Before we delve into our quantitative and qualitative methodologies, it is important to highlight the magnitude of the issue of market manipulation, as well as its impact on popular valuation techniques. To measure that, we calculated the price of various digital assets using data from hundreds of exchanges available, an approach used by several popular digital asset websites and dashboards to compute a final price. We then compared the resulting prices with our own methodology, which exclusively uses primary data from exchanges that have passed our vetting process. The differences were eye-opening.

As it turned out, the reported price of Bitcoin in that time period had an average discrepancy of 1.14%. While that might not seem much, this difference skewed Bitcoin’s market capitalization by more than $1 billion. Today, that same price discrepancy would result in a market cap difference of more than $2 billion. Beyond massive holes in valuation, the predominance of bad data can also affect ratios and other metrics, like the widely used Network Value to Transaction Value (NVT) ratio, and undermine any serious research efforts in this industry.

When we applied the same analysis to other top ten digital assets, the problem worsened considerably. Tron’s price discrepancy, for example, reached more than 35% at times this year, leading to a change in market capitalization of more than $800 million. Although such extreme discrepancies are less frequent, it is clear that the prices of all top 10 digital assets have been affected by errant prices and fabricated volume.

Because trading tends to be more centralized as we go down the list of digital assets by market capitalization, the susceptibility of smaller assets to price manipulation only increases. For this reason, the only way to avoid this multi-billion dollar hole is to meticulously scrutinize exchanges and exclude them if there is evidence that either the exchange itself, or its participants, have fabricated trades.

Regulators are Aware

Investor protection is at the forefront of financial regulatory bodies around the world, especially the Securities and Exchange Commission in the U.S. Therefore, it should come as no surprise that increased awareness of the magnitude of the pricing manipulation problem has put a damper on the approval of spot market based products, such as a Bitcoin ETF. After all, without the assurance that the reported price and volume of a digital asset reflect interactions between real buyers and sellers, it becomes incredibly difficult to assure that adequate investor protections are in place.

The SEC has cited this concern publicly on multiple occasions, and the issue of bad foundational data has led them to delay, or outright deny, all Bitcoin ETF applications to date.

We expect this to continue to be an issue until more exchanges integrate practices of nationally regulated exchanges, such as market surveillance software and sharing agreements into their operations. Nevertheless, we believe there are measures that can be taken today that can substantially increase the quality of primary data, and fulfill institutional requirements as this market matures.

The Importance of Qualitative Assessments

Wash trading is the most common form of market manipulation in traditional financial markets. It occurs when a single entity simultaneously buys and sells the same asset to simulate real market activity. In order to prevent that, exchanges have to make sure that the entities transacting are not the same, which is impossible without AML/KYC procedures. Such policies are designed to keep bad actors out from the start and provide the basis for effective market surveillance. Unsurprisingly, we found that exchanges with nonexistent, or even insufficient, AML/KYC policies were more likely to produce simulated data, and fail not only our qualitative but also quantitative assessments.

Crypto is a technology that transcends national borders. It’s also an asset class that is growing at a fast pace, so compliance with appropriate AML/KYC laws and regulations are extremely important in our qualitative process. Because the ingress point of fiat to crypto is often an exchange, the robustness of an exchange’s AML and KYC policies is an example of a qualitative factor that can reduce bad actors and increase the quality of primary data. In fact, compliance with relevant laws and regulations is one out of the three major areas in our qualitative assessment.

In our qualitative technical assessment, we evaluate an exchange’s track record of safeguarding assets as well as service uptime and the incidence of unscheduled maintenance. It is no secret that exchange security and service has been a continual concern since nearly the advent of the industry. We seek to obtain prices from exchanges that are competent service providers on both vectors of security and service. That includes institutional-grade custody, consistent service uptime, and even cold storage practices we can verify on a blockchain.

Finally, we look at an exchange’s governance and institutional factors. That includes the requirement of a transparent governance structure, clear leadership teams, business continuity plans and the aforementioned AML/KYC policies. Other factors considered include the uniformity of an exchange’s fee structure and, when deemed necessary, the soundness of its banking relationships and the existence of adequate insurance policies.

Quantitative Analysis

To complement our qualitative assessment, we devised a series of data science tests that can objectively flag suspicious trading patterns without making value judgements about an exchange, or its participants. While the sample tests described here focus on price, volume and orderbook, there are other dimensions of data we can use to probe exchanges, such as web traffic data as well as blockchain data.

Buy-Sell Permutations

Markets, as we all know, are comprised of buyers and sellers. In this data science test, we measure the permutations, or ordering, of executed buy and sell orders across all digital asset exchanges.

The graph above shows the frequency and type of orders executed in a one-hour window on Coinbase (top) and Bittrex (bottom). The green dash on the top portion of the graph signifies a buy order, whereas the red dash in the bottom represents a sell order.

We analyze trade permutations on a rolling basis, four at a time. In other words, we group trades based on their respective orders, e.g. trades 1 to 4, 2 to 5, 3 to 6, and so on.

In the example above, we highlighted four consecutive trades that took place on Gemini; one buy, one sell, another buy, and another sell. Therefore, the permutation captured in our assessment would be “bsbs”. Probabilistically, there are 16 possible permutations (42), which, as depicted below, all occurred over the course of that week.

The “bsbs” permutation (highlighted in blue) showed up with 1% frequency when compared to the distribution of all other permutations that week, and the most frequent permutation was four buys in a row (bbbb), which happened about 30% of the time, followed by four sells in a row (ssss), which happened 18% of the time.

When we applied this study to all exchanges we collect data from, a clear pattern emerged out of a core group of high-profile exchanges:

Just like the Gemini histogram depicted previously, four consecutive buys (bbbb) and four consecutive sells (ssss) were also the two most frequent buy and sell permutations that occurred on the four exchanges above. Intuitively, such patterns make sense as the market can become more favorable to buyers or sellers as it incorporates new information, which creates a natural bias for consecutive patterns.

However, we also noticed that a different pattern emerged amongst another large group of exchanges. In the chart below, we have the same permutation distribution from four exchanges we selected as examples, and unlike the distribution seen in more established exchanges, there is no identifiable bias for any particular permutation. Strangely, their permutations are evenly distributed, which is indicative of random behavior.

In fact, we compared the permutations above with that of a random coin toss and, as depicted below, their distributions are nearly identical.

It is safe to say that under no circumstances would the distribution of buy-sell permutations in a normally functioning market be naturally random. Our assumption is that these exchanges are using a random number generator, or a similar method, to simulate trades and inflate their volumes. Therefore, exchanges that have distributions that look random fail our quantitative methodology.

Correlation Testing

Our quantitative tests also assess the correlations between the data reported by exchanges of different sizes, jurisdictions, and user profiles. Given the global nature of digital asset markets, our expectation was to observe spikes and dips in volume to occur around the same time across exchanges if they are, in fact, integrated into the global marketplace.

Such expectations were met by the first four exchanges depicted in the graph above, from the top down, where spikes and dips in volume occurred roughly around the same time. However, we also notice exchanges, like the fifth one, where this pattern was not observed at all over the same time period, which is indicative of unnatural user behavior.

In order to objectively flag such outliers, our goal with this test was to track volume correlations over time and assess the consistency of market activity (or lack thereof) across different exchanges. Our methodology uses the Pearson Correlation Coefficient (⍴) to make that assessment, where:

• ⍴ measures how well two data points move together, regardless of the magnitudes of their individual values.
• ⍴ values can range from 1.00 (moving perfectly in step) to -1.00 (moving perfectly opposite).
• ⍴ values close to 0 indicate the least amount of correlation.

We applied the Pearson Correlation Coefficient to two exchanges at a time, and recorded their correlations in the matrix below. Each square represents a different coefficient value between two exchanges, and it follows a color scheme whereby negative correlation values are represented in the red spectrum, whereas neutral values are in the yellow spectrum, and positive values are in the green spectrum.

The dark green line running from the top left of the matrix to the bottom right is where each exchange is compared against itself, thereby forming a perfect correlation. We find this format to be useful, as it allows us to analyze specific groups within the matrix, such as the green concentration of highly correlated exchange pairs in its center, which experience synchronous changes in volume over the same time period.

However, we also found instances of exchanges whose volume correlated well to each other, but to no-one else. Highlighted below is an example of this discrepancy, the volume correlations between Bw and Exx. While both correlate well to each other, their reported volume figures do not correlate well to no one else in the matrix. This indicates that these exchanges are disconnected from the global marketplace and operate at odds with a highly integrated market operating 24/7.

Since subgroups as the one highlighted above seem to be at odds with the rest of global digital asset markets, we can establish correlation thresholds to filter out the noise they might be producing and limit the primary sources of data to the exchanges that truly operate as integrated marketplaces. While pairs with a p value greater than 0.5 are considered mathematically correlated, our methodology requires a coefficient of at least 0.7, which further concentrates this group of highly integrated exchanges at the center of the matrix.

Trade Lot Sizes

An exchange’s order book can also provide indisputable evidence of market manipulation occurring within. One of the order book tests used in our methodology captures the distribution of trade sizes in executed trades over a period of time. What one would expect to see in the order book of exchanges facilitating trades between real buyers and real sellers is that most trades are going to execute at less than one full bitcoin. Even when users transact large quantities of digital assets via limit or market orders, one would expect such orders to be filled in small increments as an exchange’s matching engine optimizes efficiency.

To assess that, we devised a test that measures the distribution of lot sizes across exchanges over the course of a week. We found that, by far, the most common trade lot size is 0.1 BTC, as shown in the three histograms below. In addition to common trade lot sizes, another clear pattern in exchanges that passed our qualitative vetting was the rapid decay in their trade size distribution, which rarely surpassed 2 BTC.

However, when we applied that same study to all other exchanges, a number of abnormalities emerged. The two examples below depict the same histogram of trade size and count over the same timeframe.

On the left is Oex, where 0.1 BTC is still the predominant trade size, but where there is no decay whatsoever in trade size distributions. Oddly, there is an equal frequency of trade sizes of up to 10 BTC, a pattern extremely unlikely to occur in a normally functioning market On the right is Coinbene, where both the bias for smaller trade lots and the decay in trade size are observed. Nevertheless, 0.1 BTC is still not the most common trade size and there is a high likelihood that a large portion of the trades occurring on both these exchanges is simulated.

A Continuous Process

Since the release of the Bitwise Report, we have seen a trend of data from exchanges that previously did not pass our vetting suddenly, and without a change in their reported volume, match the desired patterns of the Bitwise analysis.

For this reason, we have not included every test we perform in this post.

Our full vetting process is reviewed and enhanced quarterly, with new tests developed and limits on current tests updated. We have also implemented a waiting period: an exchange found at any time to fail either the Buy/Sell Permutations, or the Trade Lot Size tests will be removed from all data sets for a minimum period, at which time it will be reevaluated under the new set of assessments then in place.

We believe that a comprehensive approach to the vetting of primary data will ultimately increase the quality of the participants in this nascent market and expedite its maturity.

If you would like to learn more about our vetted prices and methodologies, click here to reach out.

For our free daily newsletter chocked full of the highest quality crypto news and information, sign up here.