Encrypted COM files

The main points we need to keep in mind for creating and running an encrypted COM file is the following:
  1. We need to encrypt the code
  2. We need to write some code to decrypt and execute the encrypted code
I will cover point (2) first, because once that is done it is very simple to do (1). But first I will explain the various techniques which can be used to encrypt/decrypt the code.

Encryption techniques

The two most common ways of ecrypting code is using XOR or ADD/SUB. The thing to keep in mind is that whatever method you use, it needs to be reversible. So encryption would be done as follows: 
        a = original data
        b = encryption key
        c = encrypted data

        a XOR b = c
	a ADD b = c

To decrypt we simply do the opposite: 

        c XOR b = a
        c SUB b = a

You can use other techniques such as bit shifting (ROR/ROL). I will use XOR for all examples on this page.

Decryption

The simplest way to execute an encrypted COM file is to have a loop at the start which decrypts the data and executes it. The following will XOR a block of code with the value FFh. 
        mov   si, offset encrypted_data
        mov   cx, (encrypted_data_end - encrypted_data)

dec_loop:
        xor   byte ptr [si], 0ffh
        inc   si
        loop  dec_loop           ; loop cx times

encrypted_data:
        ...
encrypted_data_end equ $

Encryption

Have a look at hello.asm. The first thing you should realize is you can compile but not execute the program as it is now. What will happen is that the XOR loop will actually encrypt the code between encrypted_data and encrypted_data_end. It will then try to executed the encrypted code, which will probably crash your computer. So how do we fix this?

The answer is, we use DEBUG! With DEBUG it is possible to load the COM file, execute the XOR loop, then write the COM file back to disk. For those who are not too familiar with DEBUG, to write a file you use 

-n [filename]                 ; set the filename
-rcx
CX 0000
:1234                         ; set number of bytes to write 1234h
-w                            ; write the file
Writing 01234 bytes

In order to know the number of bytes to write out, just look at the CX register as soon as you start debug. So this is how we would encrypt HELLO.COM: 

C:\>debug hello.com
-r                            ; check value of CX
AX=0000  BX=0000  CX=0027  DX=0000  SP=FFFE  BP=0000  SI=0000  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=0100   NV UP EI PL NZ NA PO NC
16AF:0100 BE0C01        MOV     SI,010C
-p

AX=0000  BX=0000  CX=0027  DX=0000  SP=FFFE  BP=0000  SI=010C  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=0103   NV UP EI PL NZ NA PO NC
16AF:0103 B91B00        MOV     CX,001B
-p

AX=0000  BX=0000  CX=001B  DX=0000  SP=FFFE  BP=0000  SI=010C  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=0106   NV UP EI PL NZ NA PO NC
16AF:0106 8034FF        XOR     BYTE PTR [SI],FF                   DS:010C=BA
-p

AX=0000  BX=0000  CX=001B  DX=0000  SP=FFFE  BP=0000  SI=010C  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=0109   NV UP EI PL NZ NA PO NC
16AF:0109 46            INC     SI
-p

AX=0000  BX=0000  CX=001B  DX=0000  SP=FFFE  BP=0000  SI=010D  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=010A   NV UP EI PL NZ NA PO NC
16AF:010A E2FA          LOOP    0106
-p

AX=0000  BX=0000  CX=0000  DX=0000  SP=FFFE  BP=0000  SI=0127  DI=0000
DS=16AF  ES=16AF  SS=16AF  CS=16AF  IP=010C   NV UP EI PL NZ NA PE NC
16AF:010C 45            INC     BP        <-- the first encrypted byte
-rcx
CX 0000
:27
-w
Writing 00027 bytes
-q

That's it! The HELLO.COM file is now encrypted, and will execute properly.

One final note: If you are using ADD to encrypt, you will need to use DEBUG's assemble (a) command to change the ADD instruction to a SUB instruction before writing the COM file back to disk.

© ztank
August 1998