Community post
Some weaknesses in either the DPOS algorithm or its explanation
I'm pretty horrified at the BitCoin Proof-of-Work algorithm (POW), in that I feel that it is enormously wasteful of computing resources - which translates to electrical power. I really prefer the idea of the Delegated Proof of Stake algorithm (DPOS). However, I think there are a few weaknesses in there that really should be fleshed out. And if they have - that's great - hopefully someone can put some of those explanations to paper! But if they haven't, I really do think they need to be.
The two articles I'm looking at are the Bitshares Paper, and the "Missing White Paper". My Crypto**currency** sparring partner @lukestokes recommended the latter.
Ultimately, my concerns are:
- What happens if all of the block producers collude (or a sufficiently large fraction to control production does)?
- What hidden 'centralizations' are happening that could make it so that one entity an control who ends up producing blocks?
I don't think that these two problems are even necessarily unsolvable, and I'm not even sure that they are even unsolved. I just don't see treatment of them in the papers, and I think answering these two questions is really important.
On my first reading of the BitShares paper, as I'm reading various sections, I keep asking "By who? How?" And I shouldn't have to be asking those questions. Examples:
The top N witnesses by total approval are selected.
By who? How? How do we ensure that they really are? What happens if people (nodes) disagree about who those top N are?
The witnesses are then shuffled, and ....
By who? How? How can we have a deterministic order when we 'shuffle', if every node is doing a shuffle? Unless 'shuffle' is defined some other way...(hash function based on time, maybe?)? Note: the "Missing White Paper" has a paragraph that explains this in a way that satisfies me.
There are other cases, some of which I think I already have answers for, but that critical eye when reading that first paper is important.
Next, the "Missing White Paper" -
This one completely skips the part I'm most concerned about - voting, verification, and voting-out bad actors. It goes mostly into why particular forks will be treated particular ways, and I still think I see lots of hand-waving.
I was going to start to point out that the various minority scenarios have weaknesses in them, but upon reflection I don't think it's necessary. I'll just concede that if you're a Bad Guy in the Minority, you're going to keep being outvoted by the Good Guys. That's fine, and good, and how it should work. I think that rather than going over these scenarios, we should explain more how the 'good guys' would decide to ignore your blocks. What if you are a minority producer, and you use a bunch of 'shadow nodes' - which aren't elected witnesses - to produce blocks? What if you cheat and produce blocks too fast? What if you lie on the timestamps? What if you create an entire false history back in the past, and try to integrate that? Would the longest chain still win? Would every witness, or non-Witness 'observer' of the chain have to try and confirm every single transaction back into the beginning of history?
The real part that is most important to me is "Corruption of Majority of Producers". And I think that ought to be the lede, here. What's really important is that non-Witness 'observers' are constantly looking at the blockchain, looking at the behavior of each elected Witness, and then either "calling out" bad behavior (pointing out inconsistencies on the blockchain), and also, of course, changing their votes. I'll talk more about this later, and I think this is the majority of my point.
In "Transactions as Proof of Stake (TaPoS)" -
A side effect of this process is security against long-range attacks that attempt to generate alternative chains. Individual stakeholders directly confirm the blockchain every time they transact. Over time all blocks are confirmed by all stakeholders and this is something that cannot be replicated in a forged chain.
This sounds like this could be turned around, and could be used to falsify a forged chain - just throw a bunch of transactions against blocks in it. One might even be able to do this with less than 51% of the network - just have your corrupt Witness as well as a bunch of shadow nodes creating pseudotransactions on your forged chain, and suddenly your chain might seem like the good one.
"Deterministic Producer Shuffling" - this addresses my concern about the 'Shuffle' above.
My Concerns, and the three parts I care about most
'Observers'
I think the most important feature here in DPOS is not focused on enough. While there may be 'N' active, elected Witnesses at any one time, there also needs to be a far larger number of what I'm calling 'Observers' (and, who knows, this might already be handled). Observers need to be constantly looking at the entire blockchain, as well as forks. They need to be looking for transactions that have been censored, people trying to somehow 'magic in' new currency, people trying to alter history with bad forks - and they need to yell about it. If they see something bad in the blockchain - and the blockchain is public - they can point to particular addresses, and point to particular Witnesses and be able to say "Hey! Everybody! This witness is totally screwing with things here! Vote them OUT!" Anyone would be able to look at the exact same blockchain data and make a decision for themselves. Bad-behavior Observers (trying to collude with forgers, or censor out transactions) would have to be associated with a negative reputation. Many observers would have to exist in order to handle this. I'm OK with not building in some kind of kooky reputation system for this, as I feel it would be game-able. I'm fine with just using regular, good old-fashioned web pages (Steemit pages would probably be fine, I suppose?) and good old human language to police this.
Voting
<sarcasm>Luckily, every time there is a voting event, there is always perfect agreement about the results </sarcasm>
This is really crucial, and is completely skipped in both papers. How do I know my vote has been counted? How can we all agree on who really should be the top Witnesses? What happens if different people disagree on the tallies? Where is the vote stored? How do we prevent double-voting? How do we prevent sock-puppet voting? And I could go on, and on, and on. What's to prevent a corrupt witness from bribing people - "Hey, vote for me and you'll get $x SBD every y hours!"
And the only answer I think I know is that the voting probably happens on the Blockchain somewhere? That's the only thing that would make any sense to me. Having it in some centralized data store is just....completely insane. It can't be implemented that way. I'm not even going to bother talking about the weaknesses of that scenario because they are so blatant and obvious. But the rest of my questions - I don't know the answers. And I think we need to know those answers, don't we?
Hidden Centralization
This is, perhaps, a problem more with Steemit than the DPOS algorithm itself. But if the primary way that everyone interacts with the Blockchain is through a centralized app - then...how do we know that that app isn't corrupted or compromised? And having one 'primary' or 'reference' implementation for a "Blockchain displayer app" - that effectively is a pretty centralized solution. I would hope there would be a couple of other implementations of applications that can view blog posts, comments, etc. Then, at least, we could start to look at differences between the two. Of course, there's definitely some weird business that could happen when my exact blog post is on http://steemit.com/@uberbrady as well as http://otherblockchaindisplayer.blorg/still_uberbrady. That's more about the 'canonical reference location of content' rather than anything else, and I guess the way I would try and think about that is a custom URI protocol -steemit://uberbrady/blog_post_slug - and then be able to have various different applications (or web applications) be able to parse that into a web page.
Conclusion
I still think there is absolutely a lot to like about DPOS. I think that POW has a huge advantage over it in terms of simplicity and less enforced trust; when everyone is actively competing, it really does help level the playing field. But I really do like the substantially smaller amount of waste inherent in DPOS. But I think I would feel better if there were less centralization, better documentation about voting, and a more formalized definition of the 'observer' role.
Replies (12)
I'm putting this in as a comment because I don't want to change my article after people have already read it.
There's a fourth problem and I don't know how to handle this one, either.
So let's say we have a witness who has been nothing but stellar and perfect reputation. Either the witness becomes a bad person, was already a bad person - or maybe just got hacked. Suddenly, a bad transaction is introduced. A huuuuge bad transaction, like, a million dollars worth.
How is that ever reversed? I mean, fair enough, we can now 'vote the jerk out' - but that million dollar transaction is part of the blockchain now (or a million-dollar transaction has been censored out). Somebody may have just gotten shafted pretty hard, there. What happens next?
First off, thanks for putting time and effort into actually understanding this stuff. We need more smart people like yourself actually researching and asking questions.
Secondly, I think many of the confusions you have here are in understanding the basics of blockchain technology in the first place. DPOS (in many ways) is blockchain 2.0, and I think you're confused on a lot of blockchain 1.0 concepts, based on my reading here.
Maybe this can help: Understanding Blockchain Freedom — Episode 002: What Is Cryptocurrency? (specifically, the 17 minute Blockchain 101 video I refer to there)
Or maybe just start with the source, the original 9 page Satoshi white paper: bitcoin.org/bitcoin.pdf
Many of the questions you ask here revolve around something that is not technically possible on a blockchain. If a node of the blockchain doesn't follow the agreed upon protocol, everything it does gets rejected by every other node due to the nature of what a blockchain is. It's almost like saying, "What if someone tried to send an email with custom SMTP commands?!?!" Every other SMTP server would reject the email and not send it.
Examples:
That's not possible. If a witness node tried to add a "Send X funds from account A to account B" without the proper cryptographic signatures for the accounts involved, the transaction would be immediately rejected by every node following the correct protocol.
I'll give some more examples where I think you may not yet comprehend how a blockchain works. There is no central authority determining who can do what. It's all based on the agreed protocol that all participants in the network have to follow. In order to change that protocol, a hardfork is required and that means a majority of block producers have to agree to the code change and by doing so define the new protocol everyone has to follow.
And do... what? Introduce a hardfork that includes a rule which says "all balances can be drained by @uberbrady"? Clearly, that rule would not be agreed to by stake holders and all those witnesses would be voted out as block producers. If required, the blockchain would be rewound and any transactions following the rule stake holders in the network didn't agree with would be rejected. Digging into the history of STEEM, you can find an example where a hacker injected a JavaScript hack and stole a bunch of funds. Something like this was done to protect those users and return those funds.
The only potential concern I see here is how the @steemit account (and many other Steemit, Inc related accounts with high Steem Power, such as @ned) could, in theory, vote in 11 witnesses to fork the chain in a direction the majority of other stake holders didn't agree with. Yes, it's technically possible, but it's also economic suicide. By doing so, they would destroy the value of the STEEM token as investors and stake holders would quickly lose trust and exit the platform. Hard fork 17 is an interesting example where the community disagreed with some changes Steemit, Inc wanted to make so the witnesses rejected the fork. 18 then went forward with modifications the stake holders could agree with.
By Steem Power stake holders, as defined by the blockchain protocol. Again, the very nature of a blockchain involves hashing all the history up to that point. That's what makes it immutable. All nodes following the agreed protocol ensures no block producer could just do whatever it wants. It's all built into the protocol. You're familiar enough with protocols to understand how if they aren't followed, it doesn't work. You may have a personal opinion on how HTTP should work, but all the browsers would still follow the agreed upon protocol and ignore your opinion.
Again, by the protocol. It's all open source so you could dive into the steemd C code to find the actual algorithm used here if you wanted to.
Not possible. Only witnesses can produce blocks, based on the protocol.
As I mentioned in my tweet, we currently have ~30k active users. If even one of them saw their post or vote or comment disappear, there would be hell to pay and the blockchain would store a history of who produced blocks at the point that transaction was created and why it was not included in that 3 second window block. Something like that would be absolutely damning to a witness and they would be quickly voted out if they happened to be running a custom version of the steemd code which censored certain transactions.
That said, it's a valid point that we do have to keep track of. If, within a 3 second window, a transaction isn't included, that would be bad. Once it's included, due to the nature of how blockchains work, it can't be removed as that would invalid the hashes of every transaction after that and that entire chain would be rejected by every node on the network.
Again, not possible. It's kind of funny that you suggest it might be possible. Blockchains don't allow this. :)
Again, that's the revolutionary genius of a blockchain. In this case we absolutely can agree. In fact, we have to agree. History on the blockchain is immutable. All votes are part of the blockchain as you can see on block explorers like steemd.com (or run your own).
On the blockchain.
That relates to the Byzantine Generals Problem which is one of the key innovations of blockchain technology (read the original bitcoin white paper for more on that).
This is why it's Stake-Weighted voting. It's not just a vote count. It's all about the amount of Steem Power the voter has. It costs money to obtain Steem Power. There are many very brilliant systems in place here to fight against Sybil attacks.
Nothing but free market choice and reputation. Funny thing is, it already happened when @jerrybanfield first started his witness campaign. He started sending money to people asking for votes and rewarding those who voted for him. Quickly, the community responded and many posts were written about this behavior. He apologized and has since changed his approach to realize he needs to provide real, tangible value to the community if he wants to maintain a high ranking as a witness. He's shifted tactics to spend money on advertising campaigns and commit his witness rewards to that effort.
By tools like https://steemd.com/, https://busy.org/, https://chainbb.com/ and many, many others. Some people just run their own front end locally on their laptops. I see that as the future, personally. We could all run our own custom Condensor via the open source project here: https://github.com/steemit/condenser (some are already doing this)
That's being actively discussed right now, actually. Tools like Vessel are going to use them.
Witness are still competing here as well, they are just cooperating with the users to get votes. In contrast, POW competes with users by incentivizing the miners to increase fees which hurts usability.
Geez, that was a lot to cover. I'm tempted to make this a root post so I could get paid well for the effort, but I don't want to call you out like that to my 5k+ followers, so I'll just leave this as a comment. I hope you found it helpful. :)
There are usually two different ways you can read what someone has written: assuming they're smart, or assuming they're dumb. You're doing the latter, and it hurts our ability to have a decent conversation when you do that.
I know how digital signatures work, and I understand the difference between "impossible because: crypto" versus "forbidden because: protocol". I just think the explanations need to be better. The things I feel that are not full fleshed out are things that an allegedly smart guy says he thinks are not fully fleshed-out. That's all.
Maybe it's impossible because: crypto to create a bad transaction. I mean, not just something where the signature is wrong, but something where you literally cannot say what account to transfer something into because you don't have the private key? I don't know. That's not specified. I know how this works in POW. I don't know how this works in DPOS. What if I make a bad transaction with a bad signature but a bad witness permits it? I don't know what happens. And that's the thing that I want to know.
You're hand-wavey on exactly the points I want to know about. Rewound? How the hell did that happen?
History is not a perfectly clean line, it's a gigantic tree of transactions, and we hope there's exactly one way to parse this tree into the canonical, linear list of transactions. And hopefully the explanations of how this algorithm works should explain how to do that. I think the problems I have with the explanations are solvable. I have some ideas of how I would do it if I were to make a thing, but I'm reading whitepapers that are purporting to explain the thing, and there are things that aren't explained. If I'm reading the wrong things, then point me to the right ones.
The idea that votes are weighted by activity is nice in terms of someone suddenly trying to instantiate a bunch of accounts all at once to influence a vote, but then it certainly seems dangerous that high-reputation accounts have more influence than normal folk.
Ultimately, from a protocol-perspective, I think the thing that I'm uncomfortable with is that there's only one guy producing a block at one time. It's possible to catch that guy if he does something bad, but I would prefer that the other witnesses prevent that somehow; and that those rules were built-in the protocol. Basically, I'm saying that all 'N' witnesses should try and see what they would think the block should look like, even though only one is authorized to generate the block. And if the generated block seems to be missing some things, or some signatures don't line up the way we presume they ought to, then there should be some kind of defined way to handle that.
A block is considered irreversible once 21 witnesses approved it. That means one producer and 20 verifier. Currently all witnesses run in a trustless mode, so they will verify all other produced blocks. When a witness starts the first time, it will verify the whole chain.
DPOS means delegated proof of stake, so it is determined by stake, not by aktivity. You can argue that a minority with a lot of funds has the most power, but they also have most to loose by faulting. If I own one million of all invested steem, my witness votes will be pretty influential. But if I make a wrong move, that will hurt my one million a lot.
Most of your questions seem like issues in the whitepaper, not in the Blockchain 😊
Answering again: The next witness in that round will immediately reject the block and make witness one miss a block. After 24 hours of missed blocks that witness is automatically disabled. And we have alerts for missing blocks and will by then already have multiple witnesses screaming that that witness is hacked.
(answering all this as a steem witness, bts is similar but can have subtle differences)
Good! I'd love to see a newer whitepaper with some of your answers in it. Or a blog post. or something.
These are great answers, @reggaemuffin. Anywhere you know of where I can read this level of in-depth protocol nerdery? Or would I have to go trudging through the source?
I don't think I'm well informed enough to write a whitepaper or a blog post about the topic 😅 that would mean going into detail.
But I can answer most of your questions 😊 and maybe someone with all the right knowledge makes a post about it.
Most of my knowledge is from using steem, from being a witness, from developing apps for the Blockchain and from other apps that visualize some of it. The actual steem source is something I did not yet have the time to go through.
If you have any questions, ask me and I can try to answer them and/or point you to good resources where you can read about them.
(if you like what I do and feel I am beneficial to the Blockchain, vote me as a witness 😉)
Done!!!
Thank you for your support :)
Dang it! I'm so sorry @uberbrady. I didn't see a notification about this reply, though I had checked back a couple times looking for it. Sorry for not getting back to you sooner.
I certainly don't assume you're dumb. Far from it. If my communication style implies that, I really need to improve there as that is not my intention. There's a huge difference between dumb and ignorant (currently uniformed would be a kinder way to say it).
re: rewound
Every time a node starts up, it has to replay the entire blockchain from the genesis block and veryify transactions fit the protocol. A fork in the code which evaluates those transactions which are being received from other seed nodes distributed throughout the world can then decide if a transaction is valid or not. If a transaction is received which the code determines is invalid, it will not be included in the history of the blockchain for that node. If all other nodes agree to that, then they will all have the same hashes and will all agree on what the "true" history is. This is how all blockchains work, so it's not unique to POW or DPOS. I'm not implying you're dumb by saying you don't yet understand how this works. It's complicated stuff.
Let's start from the beginning. After reading the original bitcoin white paper, you understand how the blockchain history works. The choice of DPOS over POW is just a matter of different ways to create a new block of valid transactions. The fundamental concepts of what is allowed and what is not is still the same, such as transactions have to have valid signatures before they can be included in a block and signed by the block producer. Maybe you're wanting to see the specific line of code which validates transactions to ensure the signatures match? Maybe I'm making too many assumptions that the STEEM blockchain is in fact a blockchain and follows these same validation principles?
Seems to me your question is more like: "Is STEEM a blockchain?" My answer is yes. To truly verify that, you could go through the code of the steemd client and ensure there's nothing malicious in there which would create circumstances where witnesses and nodes could somehow skip the validation process.
Again, sorry for not replying sooner. I really liked this dialogue and spent a lot of time on it. I was eagerly waiting for your reply and totally missed it. :(