Password Protected Encryption

The Problem

This page describes how to encrypt a COM file using more than just the XOR-with-some-value technique. The problem with such standard tecniques is that they are very easy to get around. If you use the method from part 1 then anyone with knowledge of using a debugger (like DEBUG.EXE) can access your original code. So instead I chose to implement a more advanced encryption engine using a password.

The first idea was to work out some 'magic value' from the password (for example, sum all characters in the password) and use that as the key for an XOR loader. But that is not secure enough, and can in fact be cracked quite easily. Some byte combinations are almost certainly going to appear in the original code (which is encrypted in the COM file). Some examples are 

	CD21        INT 21
	B8004C      MOV AX, 4C00
	CD13        INT 13

Knowledge of what the program does can help. For example if the program scans for keyboard input, that might be done through INT 16 (CD16). Let's suppose that we pick CD21 to break the encryption. The all you need to do is XOR the entire file with the bytes CD21 (you will need to do this twice because you are trying to match it with existing CD21's in the code), save the new file, and then find duplicate bytes in the new file. For example: 

	90  XOR CD = 5D
	2D  XOR 21 = 0C
	E9  XOR CD = 24    <-- Bingo! Our XOR key may be 24
	05  XOR 21 = 24    <--

	90  XOR 24 = B4
	2D  XOR 24 = 09    B409  MOV AH, 09  <-- Yes!
	E9  XOR 24 = CD
	05  XOR 24 = 21    CD21  INT 21      <-- Yes!

All such duplicates which turn up are tested on the entire file until one of them produces data which disassembles into valid code.

Essentially this means that, without too much effort, people can access the decrypted code without knowing the password.

A More Advanced Encryption Method

Before I describe the technique I used, be aware that I chose a method which would be quite time consuming to reverse engineer and break, but not too difficult to implement. This is nowhere near as secure as PGP or DES, but is easy to code!

My solution was to stay with the XOR encryption method but change it so that each byte is XORed with a different value. The main difference over the old method is that there are now two keys instead of one, and these two keys are used to generate a sequence of keys to encrypt the code with. The code for the encryption function is given below: 

; Encrypt the bytes referenced by si using ax, bx as keys. The same
; two keys can be used to decrypt the same data.
; The number of bytes is given by cx.

Encrypt		PROC NEAR
		push ax
		push cx
		push si

next_encrypt:
		test cx, cx			; check if done (cx = 0)
		jz done_encrypt
		xor byte ptr [si], al		; encrypt
		ror ax, 1
		add ax, bx			; change encryption key
		dec cx
		inc si
		jmp next_encrypt

done_encrypt:
		pop si
		pop cx
		pop ax
		ret

Encrypt		ENDP

The two instructions 

	ror ax, 1
	add ax, bx

change the key between each byte. Now all that needs to be done is to get these two "keys" from a user-supplied password.

Reading the Password

As mentioned above, two keys need to be generated from a password. However, three keys will actually be needed. The problem is we need to know before we decrypt the code whether the password is correct. If the wrong password is typed in, the above Encrypt function will generate invalid machine code, probably crashing your computer.

One way to check if the password is correct is to do a string comparision (CMPSB) between the typed in password and the correct password. Of course, doing this allows anyone with a debugger to snoop around in your code and fish out the correct password. So we can't store the original password (too easy to crack), but we can store some sort of checksum. Then, if you compare the checksum of the typed in password, and the previously saved checksum, you can expect the password to be correct if the checksums match.

Thus we need three keys. Here is my ReadPasswd function. It returns three keys in AX, BX and CX. Note that the password is not read into memory (so no need to reset the computer to clear the memory after you type in the password). 

; Read in a password and return two 'magic' numbers based on some calculation 
; of the password.
; The two values are returned in ax and bx.
; A shifted sum/xor of all the characters is returned in cx, useful for a
; checksum to print out "wrong password".

ReadPasswd	PROC NEAR
		push dx

		xor bx, bx
		xor cx, cx
		xor dx, dx

next_char:
		xor ah, ah			; read character
		int 16h				; from keyboard
		cmp al, 0dh			; enter/return -> done
		jz read_passwd_done

		xor ah, ah			; discard scancode
		add ch, al
		ror ch, 1			; ch stores shifted sum
		xor cl, al			; cl stores xor of password
		add dx, ax			; dx stores sum
		imul dl
		mov bx, ax			; bx stores product of sums
		jmp next_char

read_passwd_done:
		mov ax, dx			; ax now stores sum
		pop dx
		ret

ReadPasswd	ENDP

You may have noticed that checksum (CX) is not unique. This basically means

But, even if the checksum matches for a wrong password, the code will still be decrypted incorrectly and crash the machine. However, this seems sufficiently unlikely that it is probably not worth worring about. Finally, if any lamer who knows how to crack simple protection schemes sees your code, he will think that the checksum comparison is "the" password check. He will probably disable that check and expect the program to run perfectly... haha, what a surprise it will be when the computer crashes!
© ztank
August 1998