Virtex6 XC6VLX365T PCI-Express dev board

Around October 2021 I was searching Aliexpress and found a Chinese-made Virtex6 coprocessor board being sold by HPC FPGA Board Store. It cost around £70 at the time, and appeared to have some impressive features:

• Xilinx XC6VLX365T-1FF1156 FPGA
  • Xilinx Virtex-6 LXT series
  • 364,032 logic cells (56,880 CLB slices) and 576 DSP slices
  • 4,130 kilobits of distributed RAM and 14,976 kilobits of block RAM
  • Two PCI Express interface blocks
  • Four Ethernet MACs
  • 24 GTX transceivers
• W25Q128 SPI configuration ROM
• AT24C03 I2C EEPROM
• Can run stand-alone with DC 12V power, or powered from the PCI Express slot
  • 12V standalone power is switched on and off with a slide switch.
• JTAG on 0.1in SIL pin header or 0.1in pitch 10-pin IDC, both compatible with the Xilinx DLC10 Platform Cable II
• PCI Express x1, x4 and x8 capability (number of lanes presented to the host is configurable with DIP switches)
• 4x SFP 6.5Gbps (SFP+)
• USB TTL serial port using CH340 USB interface IC
• 8 user LEDs, 2 user pushbutton switches (plus a Reprogram/PROGRAM_B switch)
• 78-pin 0.1in user I/O header with 67x GPIOs – including a mix of LVTTL and LVDS, plus two global clocks.
• 100MHz clock provided as input to GCLK H28
• 156.25MHz GTX CLK (LVPECL)

The design files (Mediafire link) include a pin map (in the form of an Excel spreadsheet) and some demo code, but no schematic diagram and very little English documentation.

A CH340 USB serial port is fitted above the SFPs, and connects to a standard mini USB connector.

There are three switches near the SFP connectors:

• PROGRAM_B – when pushed, forces the FPGA to reset and reprogram itself.
• PIN_A9 and PIN_A10 – pull the respective FPGA pins low when pushed.

There are eight LEDs numbered D3 to D10. These are lit when the respective I/O pin (see spreadsheet) is pulled low.

A 78-pin (39 way, 2 row) 0.1in pin header carries 67 GPIOs. There is a mix of LVDS and LVTTL, as well as two LVDS GCLK (global clock) lanes.

The board is equipped with a PCI Express connector, which can operate in the x1, x4 or x8 speed modes.

Speed configuration is done via switch bank U8, near the PCIe connector:

• PCIe x1: Set the x1 switch to the right (on), leave all other switches off.
• PCIe x4: Set the x1 and x4 switches to the right (on), leave the x8 switch off.
• PCIe x8: Set all three speed select switches to the right (on).
• The fourth switch is unused and should always be switched off.

Install ISE 14.7 System Edition, and use the licence included with the demo board design files. Without a licence, ISE will only allow the XC6VLX75 device to be selected.

The design files (RAR archive) include a number of test projects:

• key_test – push the buttons to toggle the LEDs.
• led_test – walking LEDs test
• two_sfp_ibert_a_b – test SFP ports A and B?
• two_sfp_ibert_c_d – test SFP ports C and D?
• usb_ttl_com – USB serial port test
• v6_pcie_gen1x8_riffa – PCI Express device using the RIFFA framework.

First take the orange tape off the PCIe DIP switch bank, and turn on the top three switches by moving them to the right. This enables PCI Express with a maximum lane width of x8.

If the FPGA is being loaded from a 'bit' file (not the SPI Flash), then the PC must be rebooted without powering off after the FPGA is loaded. This is because the PCI Express bus is only scanned on boot.

Under Linux, it is possible to check the card's link speed and PCIe lane width:

user@thinclient:~$ lspci
...
01:00.0 Memory controller: Xilinx Corporation Device 6018
...
user@thinclient:~$ cat /sys/bus/pci/devices/0000\:01\:00.0/current_link_speed 
2.5 GT/s
user@thinclient:~$ cat /sys/bus/pci/devices/0000\:01\:00.0/current_link_width 
4

I'm testing on a HP t610 Plus thin client, which has a single PCIe x4 connector, hence the 4-lane limit. The speed of 2.5 GT/s indicates that PCI Express Gen-1 is in use.

The Linux driver in the original RIFFA repository doesn't build on later kernels, but an updated version is available. To test the card:

1. Use Git to clone https://github.com/KastnerRG/riffa.git
2. cd riffa/driver/linux && make && sudo make install
3. sudo modprobe riffa – you can use dmesg to see if the RIFFA driver was loaded.
4. cd c_c++/linux/x64/sample_app && make
5. ./testutil 0 displays the devices which RIFFA can detect.
6. ./testutil 2 0 0 10000000 performs a speed test.

It may be possible to rescan the PCIe bus and reset the card, provided the PCIe BAR configuration has not changed. This Stackoverflow thread presents some options, but I have not tried them.

The RIFFA documentation provides information on implementing a RIFFA channel interface to produce or consume data.

With RIFFA this would generally be done with the RIFFA libraries.

This Reddit FPGA subreddit post explains how to mmap the PCI resource (/sys/devices/pci<partial_device_id>/<device_id>/resourceN) and directly read and write registers.

• https://github.com/alexforencich/verilog-pcie

• The Virtex-6 on-die system monitor reports wildly incorrect readings. The chip reports temperatures of 200C and voltages of 2.8V, when the real values are around 25C and 1V.
  • 46867 - Virtex-6 FPGA ML605 Evaluation Kit - BRD and iMPACT show unexpected temperature values for ML605 – suggests checking power rails
  • 59285 - Debugging Accuracy Issues between the Temperature Diode and the System Monitor/XADC
  • 36289 - Virtex-6 System Monitor - Support for internal reference on production silicon
    • Mentions that JTAG IDCODE 2 or higher supports the internal reference
  • Virtex-6 System Monitor user guide
  • Errata sheets: 1L speed grade, other speed grades