So a couple of us were working on a reverse engineering challenge in a CTF.
We were provided with an ELF binary and an encrypted file. The goal was apparently to decrypt the file into a .PNG, the MD5SUM of which would be the flag to solve the challenge.
A cursory look at the code, either in IDA or in radare2, clearly showed that the primary purpose of the code was to XOR the entire file with the letter A.
AHA, we thought, all we have to do is an XOR. We don’t need to RE to do that. Enter xortools, a pip-installable python module. Installed, ran xor against the file, with the output as a .PNG file. Success, it looked like. The linux “file” command recognized the new file as a PNG, and we could even browse and view the image, which is exactly what we expected to see. Excitedly, we entered the MD5 of the PNG into the flag field. NOPE. Not accepted.
So it quickly became clear that the binary was doing something hinky to the file in addition to the obvious XOR, because the XOR worked and decrypted the file in a working PNG.
So Kevin and I rolled up our sleeves and got down to some RE work in radare2. I prefer IDA because it’s so much prettier, and easier to navigate and see everything, but connecting an ELF debugger to IDA is no trivial matter, and Kevin is a whiz at radare2, so off to the races we went.
First, we identified the code segment that opens, translates (via XOR) and closes the file. I’m no genius at radare2, and time constraints prevented me from fully learning assembly, but it was clear to me that the goal was to get the binary to execute that segment, and experience had showed us that earlier tricks were proving just to dance around that section of the code with evil trickery.
So we followed the desired code segment backwards, and found two decision points that would normally have an opportunity to redirect program flow. We decided to change them both in a way that would guarantee program flow in our desired direction, whether that’s changing a je/jne (jump if equal, jump if not equal) to a jmp (unconditional jump), or a NOP (no operation).
After we did that, and entered the password, program flow moved as expected, and the encrypted file was successfully decrypted to a .png. Sure enough, the md5sum of the new .png was different from the one we xor’ed manually. I put the new md5sum into the flag field, and it was ACCEPTED! Yay, we won.
But I wasn’t satisfied, I wanted to know what was different from our manually-xor’ed decryption and the one that the binary did.
So I used xxd to dump the hex output of both versions of the .png to files, then ran a diff between them.
The only difference? The very last line of the new file contained the following:
0000f380: 0a .
Meaning a single character, hex 0x0A, was appended to the file, which of course changes the checksum of the entire file without distorting the image in any way.
Let’s go back to the code and see if we can figure out why it does that.
…
Nope. No idea. Guess I’m still a noob. But we solved the challenge, and I learned some things about navigating radare2 and focusing and recognizing what’s going on in the program flow, and that’s what counts, right?