[Peggy] Reliability using Cosmos Multimsg @172k tx/day
sifchain
Go, Solidity
Friday and Saturday the peggy team rolled back to the version of the code that allowed us to send multiple transactions at once, which would theoretically give us a 20x performance boost in terms of number of tx’s per day. We did extensive testing locally under load to ensure that it functioned properly and we were not able to identify any issues. Then, we deployed to an environment in the cloud, and ran the same tests putting peggy under load. What we found were multiple dropped transactions. We were able to see invalid txs in the sifnode log, however, we were not able to retrieve those invalid transaction hashes to identify the root cause. From our current standpoint, we cannot use this multimsg version of peggy in production in its current state as it is not stable and does not move all transactions across the bridge.
**Describe the bug**
Peggy drops transactions if there are multiple transactions in an etheruem block when using the cosmos multimsg feature.
**Bug Reproduction Steps**
Check out to branch [feature/multi-msg-tx](https://github.com/Sifchain/sifnode/pull/616) and then deploy it to AWS with its own blockchain. Deploy the smart contracts to the ethereum network of choosing. Get the relayers setup. Once all the setup work is taken care of, whitelist the desired tokens in the bridgebank contract. Then spam the bridgebank with multiple lock transactions in a single ethereum block. The purpose of this is to get as many transactions into the relayer in as short a timeframe as possible. If you sent 10 transactions in a single block, we are expecting 10 transactions to be processed and go into the relayer. However, on this branch we get around 8 of the 10 transactions to actually go through. This is evidenced by the amount of tokens deposited in the bridgebank versus the amount of tokens that are received by the cosmos recipient. See token balance of this contract: https://ropsten.etherscan.io/address/0x90defdbd69a5ce54c3ecf1a9069856ccfdd739a6 and the balance of this account's pegged tokens on the sifchain testnet:
```
sifnodecli q account sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4 --node tcp://54.188.226.253:26657 --trust-node
```
There is a discrepancy as not all transactions made it across the bridge. There is a deficit of pegged tokens in acoin, c and d token in terms of locked funds in the bridgebank and the sifchain user's balance.
**Other Helpful Bug Filing Info**
Relayer tx data queried from sifnode:
```
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"sender\",\"value\":\"sif1l3dftf499u4gvdeuuzdl2pgv4f0xdtnuuwlzp8\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"success\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"sender\",\"value\":\"sif1l3dftf499u4gvdeuuzdl2pgv4f0xdtnuuwlzp8\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000cdtoken\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
"[{\"msg_index\":0,\"log\":\"\",\"events\":[{\"type\":\"create_claim\",\"attributes\":[{\"key\":\"ethereum_sender\",\"value\":\"0x1e0220B251eE648C7F3B6Fc31E6d309141f2e464\"},{\"key\":\"cosmos_receiver\",\"value\":\"sif1fpq67nw66thzmf2a5ng64cd8p8nxa5vl9d3cm4\"},{\"key\":\"amount\",\"value\":\"100000000000000000000000000000\"},{\"key\":\"symbol\",\"value\":\"dtoken\"},{\"key\":\"token_contract_address\",\"value\":\"0xfA8fC9C22C33FE62BabD5D92DD38Aa27B730d562\"},{\"key\":\"claim_type\",\"value\":\"lock\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"create_bridge_claim\"},{\"key\":\"module\",\"value\":\"ethbridge\"},{\"key\":\"sender\",\"value\":\"sifvaloper1d0q6q6afvzk3whu6cy338yth64vau78runp6u8\"}]},{\"type\":\"prophecy_status\",\"attributes\":[{\"key\":\"status\",\"value\":\"pending\"}]}]}]"
```
We queried all cosmos transaction hashes in the relayer, and none of them had any errors or anything that would suggest that there was an error in the relayer. However, when we looked at the sifnoded logs, we saw invalidTxs in the logs. We have been unable to determine any information about these transactions and what caused them to be invalid.
```
I[2021-01-31|05:46:05.271] Executed block module=state height=1657 validTxs=3 invalidTxs=1
I[2021-01-31|05:46:05.275] Committed state module=state height=1657 txs=4 appHash=953F2E9960C60856424CE8149DF900DD1D1509D049D3BBF7BD7D6CD82AFB3C4D
I[2021-01-31|05:46:10.858] Executed block module=state height=1658 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:10.862] Committed state module=state height=1658 txs=0 appHash=964A026C46793BDB914D7532C16E39491C51E87C3F70BAA852C8E7171C7924AB
I[2021-01-31|05:46:16.402] Executed block module=state height=1659 validTxs=3 invalidTxs=1
I[2021-01-31|05:46:16.406] Committed state module=state height=1659 txs=4 appHash=8DA06BFD96D9229A73A65D51FDB2C757448934D4F6FCB4D6D0E9E1CAF1A8943B
I[2021-01-31|05:46:22.017] Executed block module=state height=1660 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:22.032] Committed state module=state height=1660 txs=0 appHash=DB4502D6B39C7CBFDD5867D0DAAA4BB78EE5C09A67F501A398C616F87A898CA9
I[2021-01-31|05:46:27.779] Executed block module=state height=1661 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:27.783] Committed state module=state height=1661 txs=0 appHash=584710B4E0C2EBBDAC487EAFD91593182C6E7E84A8274821DE8E86589C833C16
I[2021-01-31|05:46:33.317] Executed block module=state height=1662 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:33.322] Committed state module=state height=1662 txs=0 appHash=AC0DE76489A4A66C90B78F4D75E1F68C3573F5761A007DE951193CCF38F67FCA
I[2021-01-31|05:46:38.872] Executed block module=state height=1663 validTxs=3 invalidTxs=1
I[2021-01-31|05:46:38.876] Committed state module=state height=1663 txs=4 appHash=6A7046AC3CCBB0F7C6FF5B2A6E05BE0C76C24C953C111FE6C64A9B45D454B6BD
I[2021-01-31|05:46:44.523] Executed block module=state height=1664 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:44.527] Committed state module=state height=1664 txs=0 appHash=EABF8D4900EF4FB8848BFB2DA95BF3F04EBE82DE29E2C4AE6E80519C6B0BFA96
I[2021-01-31|05:46:50.102] Executed block module=state height=1665 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:50.106] Committed state module=state height=1665 txs=0 appHash=615556A38D4303313411FB856EFFF20A8CC4DBC0D53AE728EDCBBCB8E5AFADE0
I[2021-01-31|05:46:55.682] Executed block module=state height=1666 validTxs=0 invalidTxs=0
I[2021-01-31|05:46:55.690] Committed state module=state height=1666 txs=0 appHash=F719762DA92BF73186FC99D14AC6BC7D482CC63C8F9F2AB7B6DE37E2AE0C2C92
I[2021-01-31|05:47:01.237] Executed block module=state height=1667 validTxs=3 invalidTxs=1
```
