Inside the Jane Street Reverse Engineering Challenge: A Month-Long Technical Journey

Jane Street periodically releases technical challenges to engage engineers and enthusiasts. One recent challenge tasked participants with reverse engineering an ASIC (Application-Specific Integrated Circuit) from its GDS file representation. The goal was to understand the chip's function and ultimately uncover a hidden password. The challenge was split into two parts: a warmup with more detailed design information and a main puzzle with minimal guidance. The participant began by exploring the GDS files, which describe the chip's physical layout, identifying familiar terms like clock (clk), reset (rst), and power signals. Using a Python library called gdstk, they analyzed the warmup puzzle's 27 elements and discovered a VCD file containing simulation data with embedded ASCII messages like "TRY AGAIN," indicating the chip encoded messages. Driven by curiosity, the participant built a custom circuit simulator and various supporting tools, including a GDS viewer and circuit design language parser. However, after several weeks, they shifted focus to existing tools and documentation, particularly the sky130 standard library used in the chip's design. This helped map geometric data to circuit inputs and outputs. Extracting the circuit's connectivity involved identifying overlapping elements across layers and coalescing wire segments. Despite the complexity—over 1,000 paths and nearly 17,000 polygons—the participant applied graph algorithms to reconstruct the circuit's structure. They translated the design into Verilog, a hardware description language, enabling simulation and validation of component behavior. The warmup puzzle revealed components like shift registers, adders, and comparators. By simulating these, the participant deduced input sequences needed to satisfy output conditions. Moving to the main puzzle, complexity increased significantly with 81 component types and nearly 10,000 elements. Performance optimizations reduced processing time, and manual implementation of new components allowed further progress. A notable discovery was an undriven wire in the circuit, which the participant reported to Jane Street. The issue was confirmed but did not affect challenge outcomes. To solve the puzzle, the participant analyzed subcircuits and timing patterns, identifying that the correct input sequence needed to produce a specific output after 120 clock cycles. Traditional forward simulation was insufficient due to input complexity, so they explored reverse simulation and constraint solving. Using the Z3 constraint solver, the participant encoded circuit behavior and output requirements as logical constraints. This approach rapidly produced valid input sequences, although debugging was challenging. Combining solver results with manual adjustments, they identified the correct input pattern. Ultimately, the participant successfully simulated the solution and confirmed the password with Jane Street. The experience highlighted the value of persistence, tool development, and leveraging constraint solvers in hardware reverse engineering. Jane Street may release new challenges in the future, offering further opportunities for engineers to test their skills.