Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transaction calldata gas cost reduction with total calldata limit #4488

Merged
merged 8 commits into from Dec 5, 2021

Conversation

vbuterin
Copy link
Contributor

Decrease transaction calldata gas cost, and add a limit of how much total transaction calldata can be in a block.

Decrease transaction calldata gas cost, and add a limit of how much total transaction calldata can be in a block.
@eth-bot
Copy link
Collaborator

eth-bot commented Nov 24, 2021

All tests passed; auto-merging...

(pass) eip-4488.md

  • passed!

(pass) assets/eip-4488/gas_and_calldata_sample.csv

  • file assets/eip-4488/gas_and_calldata_sample.csv is associated with EIP 4488; because there are also changes being made to EIPS/eip-4488.md all changes to corresponding assets are also allowed

EIPS/eip-4488.md Outdated Show resolved Hide resolved
EIPS/eip-4488.md Outdated Show resolved Hide resolved
EIPS/eip-4488.md Outdated

Hence, it's worth rethinking the historical opposition to multi-dimensional resource limits and considering them as a pragmatic way to simultaneously achieve moderate scalability gains while retaining security.

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MicahZoltu would argue to not have any external links, and potentially for this particular case to include the CSV file in the assets directory 😅

I'm happy with the external links though as they are not critical (in case they go dead), but while they are active they add some extra flavour.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You know me so well 🥲

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.
Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. At the time of writing this, 90% of all transactions taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

As @axic indicated, I think this document would be great to attach in the discussion, but I think it adds fairly minimal value long-term (to a reader in 5 years). I think just making the statement is enough, and if someone in 5 years really wants to they can dig up the data themselves or read the discussion history. If you think it really should be included, then please put it in ../assets/eip-4488/gas_and_calldata_sample.csv and link accordingly.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions (sample taken from blocks 13500000, 13501000 ... 13529000) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

Is the multi-dimentional problem also for users and wallets, not just block producers?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is only a problem for block producers.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don’t wallets estimating the fee also have this issue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EIPS/eip-4488.md Outdated

Miners will be able to continue operating with no changes except for a rule to stop adding new transactions into a block when the total calldata size reaches the maximum. However, there are pragmatic heuristics that they could add to achieve closer-to-optimal returns in such cases: for example, if a block fills up to the size limit, they could repeatedly remove the last data-heavy transaction and replace it with as many data-light transactions as possible, until doing so is no longer profitable.

## Reference Implementation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a needed section:

Suggested change
## Reference Implementation

EIPS/eip-4488.md Outdated Show resolved Hide resolved
@spartucus
Copy link

Bringing more possibilities to Rollups. :)

vbuterin and others added 2 commits November 25, 2021 07:01
Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>
Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>
Copy link
Contributor

@MicahZoltu MicahZoltu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Format/layout looks good. My only complaint is the external links. Other editors may approve to merge this with the links included, but I take a fairly hard lined stand against them.


## Rationale

A natural alternative proposal is to decrease `NEW_CALLDATA_GAS_COST` without adding a limit. However, this presents a security concern: today, the average block size [is 60-90 kB](https://etherscan.io/chart/blocksize), but the _maximum_ block size is `30M / 16 = 1,875,000` bytes (plus about a kilobyte of block and tx overhead). Simply decreasing the calldata gas cost from 16 to 3 would increase the maximum block size to 10M bytes. This would push the Ethereum p2p networking layer to unprecedented levels of strain and risk breaking the network; some previous live tests of ~500 kB blocks a few years ago had already taken down a few bootstrap nodes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A natural alternative proposal is to decrease `NEW_CALLDATA_GAS_COST` without adding a limit. However, this presents a security concern: today, the average block size [is 60-90 kB](https://etherscan.io/chart/blocksize), but the _maximum_ block size is `30M / 16 = 1,875,000` bytes (plus about a kilobyte of block and tx overhead). Simply decreasing the calldata gas cost from 16 to 3 would increase the maximum block size to 10M bytes. This would push the Ethereum p2p networking layer to unprecedented levels of strain and risk breaking the network; some previous live tests of ~500 kB blocks a few years ago had already taken down a few bootstrap nodes.
A natural alternative proposal is to decrease `NEW_CALLDATA_GAS_COST` without adding a limit. However, this presents a security concern: at the time of writing this, the average block size is 60-90 kB, but the _maximum_ block size is `30M / 16 = 1,875,000` bytes (plus about a kilobyte of block and tx overhead). Simply decreasing the calldata gas cost from 16 to 3 would increase the maximum block size to 10M bytes. This would push the Ethereum p2p networking layer to unprecedented levels of strain and risk breaking the network; some previous live tests of ~500 kB blocks a few years ago had already taken down a few bootstrap nodes.

I think there may be value in including this link in the discussion thread, but in 5-10 years the link will likely be a 404 and it almost certainly won't contain useful data for the context of someone reading this EIP at that time so it shouldn't be included in the EIP itself.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the right place for this link is as a comment linking to the statement -- directly on this PR.

Presuming the PR is merged, we don't need to go back and verify the block-size statement again, since potentially new sites would provide that data when a future reader looks it up.

EIPS/eip-4488.md Outdated

The decrease-cost-and-cap proposal achieves most of the benefits of the decrease, as rollups are unlikely to _dominate_ Ethereum block space in the short term future and so 1.5 MB will be sufficient, while preventing most of the security risk.

Historically, the Ethereum protocol community has been suspicious of multi-dimensional resource limit rules (in this case, gas and calldata) because such rules greatly increase the complexity of the block packing problem that proposers (today miners, post-merge validators) need to solve. Today, proposers can generate blocks with near-optimal fee revenue by simply choosing transactions in highest-to-lowest order of priority fee. In a multi-dimensional world, proposers would have to deal with multi-dimensional constraints. [Multi-dimensional knapsack problems](https://en.wikipedia.org/wiki/Knapsack_problem#Multi-dimensional_knapsack_problem) are much more complicated than the single-dimensional equivalent, and well-optimized proprietary implementations built by pools may well outperform default open source implementations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Historically, the Ethereum protocol community has been suspicious of multi-dimensional resource limit rules (in this case, gas and calldata) because such rules greatly increase the complexity of the block packing problem that proposers (today miners, post-merge validators) need to solve. Today, proposers can generate blocks with near-optimal fee revenue by simply choosing transactions in highest-to-lowest order of priority fee. In a multi-dimensional world, proposers would have to deal with multi-dimensional constraints. [Multi-dimensional knapsack problems](https://en.wikipedia.org/wiki/Knapsack_problem#Multi-dimensional_knapsack_problem) are much more complicated than the single-dimensional equivalent, and well-optimized proprietary implementations built by pools may well outperform default open source implementations.
Historically, the Ethereum protocol community has been suspicious of multi-dimensional resource limit rules (in this case, gas and calldata) because such rules greatly increase the complexity of the block packing problem that proposers (today miners, post-merge validators) need to solve. Today, proposers can generate blocks with near-optimal fee revenue by simply choosing transactions in highest-to-lowest order of priority fee. In a multi-dimensional world, proposers would have to deal with multi-dimensional constraints. Multi-dimensional knapsack problems are much more complicated than the single-dimensional equivalent, and well-optimized proprietary implementations built by pools may well outperform default open source implementations.

Someone who wants to know what the Multi-dimensional knapsack problem is can search the internet for it just like they could for any other term they are unfamiliar with in this document. Separately (and unrelated to this PR), Wikipedia is pure evil now.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @MicahZoltu just out of curiosity, why do you think that about Wikipedia?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Their editorial "staff" (technically volunteers I think) has been captured by groups with strong biases and an unwillingness to present a factual and neutral view that doesn't align with their personal world view.

Try to make technical corrections to cryptocurrency related Wikipedia pages if you want to feel what it is like personally. As an example, a while back I tried to make corrections to a page about an open source dapp and all of my edits (purely technical corrections based on the source code) were rejected because mainstream media hadn't reported on them, and thus they claimed I was synthesizing information.

EIPS/eip-4488.md Outdated

Hence, it's worth rethinking the historical opposition to multi-dimensional resource limits and considering them as a pragmatic way to simultaneously achieve moderate scalability gains while retaining security.

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.
Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. At the time of writing this, 90% of all transactions taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

As @axic indicated, I think this document would be great to attach in the discussion, but I think it adds fairly minimal value long-term (to a reader in 5 years). I think just making the statement is enough, and if someone in 5 years really wants to they can dig up the data themselves or read the discussion history. If you think it really should be included, then please put it in ../assets/eip-4488/gas_and_calldata_sample.csv and link accordingly.

EIPS/eip-4488.md Outdated Show resolved Hide resolved
EIPS/eip-4488.md Outdated Show resolved Hide resolved

Rollups are in the short and medium term, and possibly the long term, the only trustless scaling solution for Ethereum. Transaction fees on L1 have been very high for months and there is greater urgency in doing anything required to help facilitate an ecosystem-wide move to rollups. Rollups are significantly reducing fees for many Ethereum users: [l2fees.info](https://l2fees.info/) frequently shows Optimism and Arbitrum providing fees that are ~3-8x lower than the Ethereum base layer itself, and ZK rollups, which have better data compression and can avoid including signatures, have fees ~40-100x lower than the base layer.

However, even these fees are too expensive for many users. The long-term solution to the long-term inadequacy of rollups by themselves has always been [data sharding](https://github.com/ethereum/consensus-specs#sharding), which would add ~1-2 MB/sec of dedicated data space for rollups to the chain. However, data sharding will still take a considerable amount of time to finish implementing and deploying. Hence, a short-term solution to further cut costs for rollups, and to incentivize an ecosystem-wide transition to a rollup-centric Ethereum, is desired.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
However, even these fees are too expensive for many users. The long-term solution to the long-term inadequacy of rollups by themselves has always been [data sharding](https://github.com/ethereum/consensus-specs#sharding), which would add ~1-2 MB/sec of dedicated data space for rollups to the chain. However, data sharding will still take a considerable amount of time to finish implementing and deploying. Hence, a short-term solution to further cut costs for rollups, and to incentivize an ecosystem-wide transition to a rollup-centric Ethereum, is desired.
However, even these fees are too expensive for many users. The long-term solution to the long-term inadequacy of rollups by themselves has always been data sharding, which would add ~1-2 MB/sec of dedicated data space for rollups to the chain. However, data sharding will still take a considerable amount of time to finish implementing and deploying. Hence, a short-term solution to further cut costs for rollups, and to incentivize an ecosystem-wide transition to a rollup-centric Ethereum, is desired.

I'm generally OK with linking to other Ethereum specifications, but I prefer to avoid it when possible in case those specifications move to a new location. In this case, I think that there is some value in linking here, but not enough to overcome my personal burden of need to warrant an external link. Understanding data sharding isn't critical to understanding this EIP and it is a relatively easy term to search the internet for if you are uncertain about it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to disagree on this suggested removal, as the official nature of this link isn't an "external" link in the same way that linking to a different site would be; perhaps directly linking to a specific commit's variant of the standard would help?

Additionally, the URL itself has value for future readers, as it gives them some useful keywords for finding the right section of a hypothetically moved spec.


Users will be able to continue operating with no changes.

Miners will be able to continue operating with no changes except for a rule to stop adding new transactions into a block when the total calldata size reaches the maximum. However, there are pragmatic heuristics that they could add to achieve closer-to-optimal returns in such cases: for example, if a block fills up to the size limit, they could repeatedly remove the last data-heavy transaction and replace it with as many data-light transactions as possible, until doing so is no longer profitable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is indeed the optimal strategy for miners - to drop rollup transactions in favor of more execution-oriented transactions. Isn't there a risk that it'll actually hurt rollups?

At high congestion times (e.g. big NFT sales) rollup transactions will be constantly dropped, and they'll have to compensate for the lack of execution gas by paying a higher total fee. Theoretically it should drive rollup gas fees to the current cost, except that it also limits the calldata size. The additional constraint might require them to pay an even higher fee to outbid other rollups competing on the same calldata space.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend making this comment in the discussions-to link found in the EIP header. PR comments are very transient and not a good place for discussions about the EIP content.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@stevengcook
Copy link

Add a rule that a block is only valid if sum(len(tx.calldata) for tx in block.txs) <= BASE_MAX_CALLDATA_PER_BLOCK + len(block.txs) * CALLDATA_PER_TX_STIPEND.

Would it not be easier (and slightly less variable max calldata per block) if only transactions with more than CALLDATA_PER_TX_STIPEND are counted against BASE_MAX_CALLDATA_PER_BLOCK?
Assuming 300 is the stipend per block, transactions with <= 300 calldata don't provide extra room for transactions that use > 300 calldata.

@MicahZoltu
Copy link
Contributor

@stevengcook I recommend asking this question in the discussions-to forum thread (at top of EIP).

@stevengcook
Copy link

@stevengcook I recommend asking this question in the discussions-to forum thread (at top of EIP).

my bad, thanks!

Co-authored-by: Micah Zoltu <micah@zoltu.net>
Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>
@alita-moore
Copy link
Contributor

A critical exception has occurred (cc @alita-moore): Message: not found

I will fix this today, btw

@alita-moore alita-moore closed this Dec 2, 2021
@alita-moore alita-moore reopened this Dec 2, 2021
@lightclient lightclient closed this Dec 5, 2021
@lightclient lightclient reopened this Dec 5, 2021
@eth-bot eth-bot enabled auto-merge (squash) December 5, 2021 23:42
@eth-bot eth-bot merged commit 0226f2f into master Dec 5, 2021
@eth-bot eth-bot deleted the vbuterin-patch-1 branch December 5, 2021 23:45
PhABC pushed a commit to PhABC/EIPs that referenced this pull request Jan 25, 2022
…hereum#4488)

* Transaction calldata gas cost reduction with total calldata limit

Decrease transaction calldata gas cost, and add a limit of how much total transaction calldata can be in a block.

* Minor edits

* Fixed parameter name

* Update EIPS/eip-4488.md

Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>

* Update EIPS/eip-4488.md

Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>

* Apply suggestions from code review

Co-authored-by: Micah Zoltu <micah@zoltu.net>
Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>

* remove wikipedia link

* bring csv into eip assets

Co-authored-by: Alex Beregszaszi <alex@rtfs.hu>
Co-authored-by: Micah Zoltu <micah@zoltu.net>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Copy link

@hydrogenbond007 hydrogenbond007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rollup builders are gonna love this + some chains that don't support blob txns are using this proposal for research!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet