Synchronize a cluster of Red Pitayas

In its standard configuration, the Red Pitaya uses an on-board 125 MHz crystal to feed the 125 MSPS ADC and the 125 MSPS DAC. In this post, we present how to synchronize the outputs of multiple Red Pitayas on the same clock using the SATA connector (daisy-chain) available on the Red Pitaya.

The figure above describes the standard clocking configuration of the Red Pitaya. The FPGA receives from the ADC LTC2145 a 125 MHz clock that is synchronous to a crystal oscillator. This clock is used as an input for a phase-locked loop (PLL) inside the FPGA. The DAC (AD9767) operates in double data rate and thus needs to be fed by a 250 MHz clock from the PLL.

We want to synchronize the DACs of 4 Red Pitayas on the same 125 MHz clock. One of the Red Pitayas will be clocked by its 125 MHz crystal (Red Pitaya 2 in the figure below) and will transfer the clock to the next Red Pitaya (and so on) through the SATA connectors. Note that in this design, only the ADC of Red Pitaya 2 will work correctly. Synchronizing the ADCs requires some soldering (link).

Clocking wizard

The PLL can be configured with the Vivado clocking wizard ([1]). Here we select the MMCM (mixed-mode clock manager) primitive, which is very similar to the PLL but has more capabilities such as dynamic phase shift. The differential clock input CLK_IN1_D will be connected to the SATA input clock sata_clk_in and CLK_IN2_D will be connected to the crystal clock crystal_clk. The pin clk_in_sel is used to select between the two input clocks.

Below is a snapshot of the MMCM instantiated in Vivado IP integrator. The yellow paths are used for run-time configuration of the MMCM. The MMCM input pins psclk, psen and psincdec are used to adjust dynamically the phase shift between the input clock and the output clock of the MMCM. This allows to adjust the phase with increments of $ 1/(56 \times F_{VCO}) $, where $ F_{VCO} $ is the frequency of the voltage-controlled oscillator of the MMCM. In our case, the VCO frequency is 1 GHz, which means that we can adjust the phase down to a precision of 18 ps.

Implementation

The final block design is shown in the figure below. The purple path corresponds to the DDS (Direct Digital Synthesis) that outputs a sine wave of user-defined frequency to the Red Pitaya DAC. The first differential pair of the SATA connectors is used to transfer data along with the clock.

Our 4 Red Pitayas can be controlled from a unique Python script as below.

# Define the IP addresses of the 4 Red Pitayas
hosts = ['192.168.1.14', '192.168.1.5', '192.168.1.13', '192.168.1.6']
clients = []
drivers = []

for i, host in enumerate(hosts):
    clients.append(connect(host, 'cluster', restart=False))
    drivers.append(Cluster(clients[i]))

for driver in drivers:
    driver.set_freq(10e6) # 10 MHz sine
    driver.set_clk_source('crystal')

# The three last Red Pitaya are clocked through the SATA connectors
for i in [1,2,3]:
    drivers[i].set_clk_source('sata')

# Phase shift the second Red Pitaya
for i in range(10000):
    drivers[1].phase_shift(1)
    print(i)
    time.sleep(0.01)

The 4 10 MHz sine waves generated by each Red Pitaya have exactly the same frequency and their phase relationship stays constant over time. The phase relationship can be adjusted thanks to the dynamic phase shift capability of the MMCM.

[1]: 7 Series FPGAs Clocking Resources, Xilinx User Guide, 2016

FPGA design source code

https://github.com/Koheron/koheron-sdk/tree/master/examples/red-pitaya/cluster