Category: Crypto

Crypto warning: I am not an expert. I don’t claim to be an expert. Don’t “ekshually” me. I figured something out, I’m sharing it with other people, some of whom are stupider than me, in case it’s useful to someone some day. I also may have the facts wrong. I’m not a Defcoin insider either.

So, a while back, Coinomi started supporting Defcoin. YAY.

No, they didn’t allow exchanging it with fungible cryptocurrencies. BOO.

But at least we had a nice phone-based wallet to send, receive, etc. at the con. YAY.

But then it stopped working. Was it last year? I’m fuzzy on the whens. BOO.

Apparently there was some legacy shit in there, maybe it had to do with finding other clients, a hard-coded list, I don’t remember.

Anyhow, sometime fairly recently, it started working again. YAY.

But not for me. BOO.

Turns out, the coinomi wallet I had set up had a LOT of transactions. And I think it was too much for the app to handle. Or maybe something got mangled somehow in the blockchain. Someone explained it once, but you know, I forgot it right away. Like a goldfish.

Look, it’s not about the “money.” I’m not even sure how many Defcoin were in that wallet anymore. Thousands? Value of 0? It’s about wanting a working Defcoin wallet on my phone. And sure, I could have just deleted it and started fresh, but I wanted to learn something and get what I could recover from it.

SO, here are the steps I took and how that worked out for me.

1. Set up a new Defcoin instance in Coinomi. Yes, it’ll allow you to add a coin twice.
2. For my next trick, I wanted to sweep the old wallet into the newly-added instance.
3. Coinomi doesn’t exporting private keys that allow you to do so, but you can use your mnemonic recovery phrase to generate that. Download this to your local machine, or better yet, put it on an air-gapped machine. Because you’re going to be typing your mnemonic phrase into it, and anyone that gets their grubby paws on that can steal your fundage. https://iancoleman.io/bip39/
4. OK, so load that page up locally on your safely air-gapped system, then type your mnemonic phrase into the BIP39 mnemonic phrase field.
5. Set the “COIN” to “Defcoin.”
6. In the Derivation Path section, go to the BIP44 tab.
7. From your broken Coinomi defcoin wallet, look at Account Details. Get the Derivation Path from there. Mine was m44h/1337h/0h. Type that, without the h characters, into the derivation path field.
8. Now look at your receive addresses in the old account. Including previous receive addresses. You should see those addresses down below in the extractor, in the “Derived Addresses” section. One by one, take the private keys associated with those addresses, and throw them into the “Sweep Wallet” private key field in the NEW Defcoin instance in Coinomi.
   
   This worked for me for several of my receive addresses, leading to thousands of Defcoin recovered into the new wallet. However, the one I used so often that I recognized the preamble, took way longer for Coinomi to attempt to sweep it, and it came back with “The private key does not contain any funds,” which I believe is bullshit, I just think there are, once again, too many transactions for Coinomi to handle. But at least now I have a working wallet that can send and receive funds. I just may have lost those funds associated with the largest receive address in my wallet. Perhaps I’ll try again in the future.

Another option is to try to sweep it into a native Defcoin-QT wallet, but that would require converting the private key to WIF format, which might be a bit beyond me at this hour.

Follow along in my journey of fixing a broken NOMP/MPOS Defcoin mining pool. It wasn’t a public pool, it was my own personal solo mining pool. The idea was that it would eventually become public, but you know how it is, sometimes it takes time to get around to doing things. Doing something for me is easy, doing it for public requires much more careful thought and planning.

Careful thought and planning that I wasn’t executing last year, sometime between February and April, when I haphazardly ran an apt upgrade on the Ubuntu 18.04 VM that was running my pool. I didn’t think anything of it. It was a busy time. I was in Vegas for a while in February, then I came home and went to BSides Nova, then the world shut down and mining Defcoin was just not on my mind.

I noticed that my wallet wasn’t getting fatter, so I logged in to take a look, and realized it was 100% out of space. The shares table was 3GB. It wasn’t important at the time, so I abandoned it in place.

Cut to this week, when bashNinja and others are talking about doing some work on Defcoin. Pools are popping up, people are getting excited again, there’s talk of forks, and I’m right there paying attention, because sure, I want my pool up and running again. But man, I’m not looking forward to figuring out this software made of black magic and rickety scaffolding and held together with government cheese. I barely got it running the first time, I clearly didn’t understand it.

So, reluctantly, I started digging. First, my defcoin core wallet is not talking to any peers. It only has one peer address and it can’t connect. Well, it has been a year. I asked on bashninja’s discord about that, and got a quick and easy response. I was pointed to a post in /r/defcoin that contained a list of peers that can be manually added via the defcoin-qt debug console window. Once I did that, it started to talk to peers again, and began to wriggle its way towards 2021 on its own time.

Second, the shares table. Nothing can work with that table in that state, everything’s just running too damn slow. 17 million rows. So…

Let’s clear that table. At this point I have no idea whether it will prevent the rest of the system from working. [Keep in mind that I never gained a full understanding of how the system is strung together, I just got it working and let it go. So at this point, I’m reverse engineering something I slapped together myself.] But in case I need it, I’ll back it up. So… create table shares_manual_backup like shares; insert into shares_manual_backup select * from shares; Then, once I confirmed everything copied, delete every row from shares. This allowed me to navigate, and allowed the WebGUI to respond again. I needed that, there’s valuable troubleshooting info hiding in there.

So browsing around the GUI, I see that all the cron jobs have been disabled. It took me a while to remember where to find and fix that. I don’t know why an interface wasn’t created for it. How it works, I learned, is that if one of the cron jobs and their subtasks fail, they update or add a row in the monitoring table to indicate that the job is disabled, then they no longer run from cron, forcing the administrator to address the underlying issue before it gets worse.

I tried enabling them and running them, they just revert back to disabled. So I dug around to find where MPOS logs results of those cron jobs, and I found them. /home/(username)/mpos/logs/(jobname)/log_(date)etc. I found very strange results in those log files. Problems with scripts that I hadn’t changed. Curiouser and curiouser.

So again this took a while, but eventually I happened up on a clue. A script failing because a command had been deprecated in PHP 8. So now it’s starting to dawn on me that my update might have caused this. Also, it’s having trouble finding memcached, which I know is installed. I don’t quite understand, until…

OK, I’ll add a phpinfo file to the public-facing web area of MPOS. Go to it. Sure enough, no memcached. But wait. This says we’re running PHP 7.3. How can this be? Back to command-line. php -v shows PHP 8.0. What is this trickery??? OK. Since the problem is clearly in the command-line, because the cron jobs are failing, let’s try backing this version down to PHP 7.3. This can be done with update-alternatives.

That worked. Now we’re getting different errors.

021-04-14 18:53:49 - ERROR --> Failed to update share ID in database for block 1273177: SQL Query failed: 2006
2021-04-14 18:53:49 - ERROR --> Failed to update worker ID in database for block 1273177: SQL Query failed: 2006
2021-04-14 18:53:49 - ERROR --> Failed to update share count in database for block 1273177: SQL Query failed: 2006
2021-04-14 18:53:49 - CRIT --> E0005: Unable to fetch blocks upstream share, aborted:Unable to find valid upstream share for block: 1273178
2021-04-14 18:53:49 - INFO --> |    23103 |    1273178 |           24.75 |            |                           | []              |                 |          any_share |
2021-04-14 18:53:49 - ERROR --> Failed to update share ID in database for block 1273178: SQL Query failed: 2006
2021-04-14 18:53:49 - ERROR --> Failed to update worker ID in database for block 1273178: SQL Query failed: 2006
2021-04-14 18:53:49 - ERROR --> Failed to update share count in database for block 1273178: SQL Query failed: 2006

OK, this makes sense. Of course it can’t associate share IDs with blocks, I’ve wiped out the share table! So let’s look closer at the share table, because I’m really hesitant to dump 17 million records back in. Looking closer at the data, the way it associates a share ID with a block is the “solution” field in the shares table, which maps to the “blockhash” field in the blocks table. A couple of quick count queries reveal that of the 17 million records in the shares table, currently relocated, fewer than 7,000 contain a populated solutions field. Those are the shares that resulted in a blockhash. So, on a hunch, I select just those rows back into shares and run the findblocks command again. Lo and behold, it’s not failing. It’s taking its time, though. About two seconds for every three records. So roughly this “fix,” assuming it works, will take a while.

I let it run for a while, and then I tentatively give the pps-payout script a poke, since that’s another one that was failing instantly because it wasn’t finding any shares that matched its criteria. Sure enough, it’s able to chew on the data that findblocks is now fixing. Good.

So the way the scripts are re-enabled is, you fix the underlying problem, then run the script with the -f argument. If it succeeds, it re-enables the cron job. So it’s important to check that, because any problem can cause a cascade of further problems that eventually kill the system.

I probably won’t know until midnight tonight whether I’m finished with my NOMP/MPOS deep dive, but I will sleep well knowing that I’ve taken it far from the broken state it was in, and I’ve learned a lot along the way. Oh, and I documented everything I found in my personal Gitlab issues and Wiki for the project, so even if I unlearn it, it’ll be less painful next time.