Difference between revisions of "RZ-G/RZG2 pcie ep"

From eLinux.org
Jump to: navigation, search
m (COPY test)
(One intermediate revision by the same user not shown)
Line 12: Line 12:
 
DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10
 
DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10
  
== 🛠️ Cable Requirements for RZ/G2{HMN} Rev.4.0 ==
+
== 🛠️ HW Requirements ==
 +
=== 🛠️ Cable Requirements for RZ/G2{HMN} Rev.4.0 ===
  
= [[File:CablerequirementsImage.png|Cable Requirements for RZ/G2E RevA/B]] =
+
[[File:CablerequirementsImage.png|600px|Cable Requirements for RZ/G2E RevA/B]]
 +
<br clear="left" />
 +
<br/>
  
== 🛠️ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B ==
+
=== 🛠️ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B ===
  
 
* Remove U11 (9fgv0441) from the board
 
* Remove U11 (9fgv0441) from the board
Line 22: Line 25:
 
* Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).
 
* Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).
  
= [[File:Rzg2ehwmodpcieep.png|frame|left|Hardware Modifications for RZ/G2E RevA/B]] =
+
[[File:Rzg2ehwmodpcieep.png|left|Hardware Modifications for RZ/G2E RevA/B]]
 +
<br clear="left" />
 +
<br/>
 +
<br/>
 +
[[File:Siliconlinuxpcieepcable.png|700px|left|Cable Requirements for RZ/G2E RevA/B]]
 +
<br clear="left" />
  
= [[File:Siliconlinuxpcieepcable.png|frame|left|Cable Requirements for RZ/G2E RevA/B]] =
+
== Other Requirements ==  
 
+
=== 📄 Host PC Tools ===
== 📄 Host PC Tools ==
 
 
* Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.
 
* Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.
  
== 📄 Preconditions ==
+
=== 📄 Preconditions ===
 
* Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
 
* Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
 
* Hardware modifications required for the board to work as endpoint as per the requirements above.
 
* Hardware modifications required for the board to work as endpoint as per the requirements above.
 
* pcitest utility should be part Rootfs/NFS for board working as RC.
 
* pcitest utility should be part Rootfs/NFS for board working as RC.
 +
[[File:20191025_115443.jpg|500px|left|PCIe connection]]
 +
<br clear="left" />
  
= [[File:20191025_115443.jpg|frame|left|PCIe connection]] =
+
== 📄 Configuration ==
 
+
=== Linux Driver Configuration for EP ===
== 📄 Linux Configuration ==
 
= Driver Configuration for EP =
 
 
The following config options has to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.
 
The following config options has to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.
  
Line 46: Line 53:
 
</pre>
 
</pre>
  
 
+
=== 📄 PCIe EP DTS configuration ===
= 📄 PCIe EP DTS configuration =
 
 
By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.
 
By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.
  
= RZ/G2H HiHope board in EP mode: =
+
==== RZ/G2H HiHope board in EP mode: ====
  
 
<pre>
 
<pre>
Line 71: Line 77:
 
</pre>
 
</pre>
  
= RZ/G2M HiHope board in EP mode: =
+
==== RZ/G2M HiHope board in EP mode: ====
  
 
<pre>
 
<pre>
Line 92: Line 98:
 
</pre>
 
</pre>
  
= RZ/G2N HiHope board in EP mode: =
+
==== RZ/G2N HiHope board in EP mode: ====
  
 
<pre>
 
<pre>
Line 113: Line 119:
 
</pre>
 
</pre>
  
= RZ/G2E EK874 board in EP mode: =
+
==== RZ/G2E EK874 board in EP mode: ====
  
 
<pre>
 
<pre>
Line 132: Line 138:
 
</pre>
 
</pre>
  
= PCIe RC DTS configuration =
+
==== PCIe RC DTS configuration ====
  
 
By default PCIe controller is enabled in root complex mode in the DTS.
 
By default PCIe controller is enabled in root complex mode in the DTS.
Line 149: Line 155:
 
</pre>
 
</pre>
  
= Using the pci-epf-test function driver =
+
=== Using the pci-epf-test function driver ===
 
The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported
 
The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported
  
Line 158: Line 164:
 
* Copy tests
 
* Copy tests
  
= Creating pci-epf-test device =
+
=== Creating pci-epf-test device ===
 
PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used
 
PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used
 
<pre>
 
<pre>
Line 185: Line 191:
 
</pre>
 
</pre>
  
= Creating pci-epf-test device =
+
=== Creating pci-epf-test device ===
 
The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.
 
The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.
  
Line 201: Line 207:
 
</pre>
 
</pre>
  
= Configure MSI Interrupt =
+
=== Configure MSI Interrupt ===
 
Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them.
 
Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them.
 
<pre>
 
<pre>
Line 208: Line 214:
 
Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.
 
Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.
  
= Binding pci-epf-test device to a EP controller =
+
=== Binding pci-epf-test device to a EP controller ===
 
In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.
 
In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.
 
<pre>
 
<pre>
Line 214: Line 220:
 
</pre>
 
</pre>
  
= Starting the EP device =
+
=== Starting the EP device ===
 
In order for the EP device to be ready to establish the link, the following command should be given.
 
In order for the EP device to be ready to establish the link, the following command should be given.
 
<pre>
 
<pre>
Line 221: Line 227:
 
Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).
 
Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).
  
== PCIe RC Setup ==
+
== PCIe RC ==
 +
The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesn’t support hot plug.
  
The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesn’t have hot plug support.
+
=== Endpoint Test function device ===
 +
pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.
 +
<pre>
 +
# insmod pci_endpoint_test.ko
 +
</pre>
  
lspci output
+
=== Verification of PCIe EP on RC using lspci ===
 
<pre>
 
<pre>
 
root@salvator-x:~# lspci -vv
 
root@salvator-x:~# lspci -vv
Line 331: Line 342:
 
</pre>
 
</pre>
  
= Using the Endpoint Test function device =
+
== pcitest ==
pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.
+
=== Building pcitest ===
<pre>
 
# insmod pci_endpoint_test.ko
 
</pre>
 
 
 
== Building pcitest ==
 
 
<pre>
 
<pre>
 
# cd <kernel-dir>
 
# cd <kernel-dir>
Line 345: Line 351:
 
'''Note:''' The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.
 
'''Note:''' The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.
  
= pcitest options =
+
=== pcitest options ===
 
<pre>
 
<pre>
 
$ pcitest  -h
 
$ pcitest  -h
Line 364: Line 370:
 
</pre>
 
</pre>
  
= BAR test =
+
=== BAR test ===
 
<pre>
 
<pre>
 
$ pcitest -b 0
 
$ pcitest -b 0
Line 376: Line 382:
 
'''💡 Note:''' BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable
 
'''💡 Note:''' BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable
  
= Legacy interrupt test =
+
=== Legacy interrupt test ===
 
<pre>
 
<pre>
 
$ pcitest -i 0
 
$ pcitest -i 0
Line 384: Line 390:
 
</pre>
 
</pre>
  
= MSI interrupt test =
+
=== MSI interrupt test ===
 
<pre>
 
<pre>
 
$ pcitest -i 1
 
$ pcitest -i 1
Line 398: Line 404:
 
</pre>
 
</pre>
  
= READ test =
+
=== READ test ===
 
<pre>
 
<pre>
 
$ pcitest -r 1024
 
$ pcitest -r 1024
Line 404: Line 410:
 
</pre>
 
</pre>
  
= WRITE test =
+
=== WRITE test ===
 
<pre>
 
<pre>
 
$ pcitest -w 1024
 
$ pcitest -w 1024
Line 410: Line 416:
 
</pre>
 
</pre>
  
= COPY test =
+
=== COPY test ===
 
<pre>
 
<pre>
 
$ pcitest -c 1024
 
$ pcitest -c 1024

Revision as of 11:17, 6 January 2021

🐧 Renesas RZ/G2 PCIe EP

Introduction

The PCIe controller IP in RZ/G2 is capable of operating either in Root Complex mode (host) or Endpoint mode (device). When operating in endpoint mode, the controller can be configured to be used as any function depending on the use case. ("Test endpoint" is the only PCIe EP function supported in Linux kernel right now). The endpoint driver supports both MSI and legacy interrupt modes.

Source Code

Driver and DT (PCIe EP nodes in SoC DTSI) additions have been merged into upstream and are part of the Linux 5.10 release.

PCIe EP driver - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-rcar-ep.c?h=v5.10

DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10

🛠️ HW Requirements

🛠️ Cable Requirements for RZ/G2{HMN} Rev.4.0

Cable Requirements for RZ/G2E RevA/B

🛠️ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B

  • Remove U11 (9fgv0441) from the board
  • Use a wire to connect the pads from pin 13 (DIF0) to pin 18 (DIF1).
  • Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).
Hardware Modifications for RZ/G2E RevA/B




Cable Requirements for RZ/G2E RevA/B


Other Requirements

📄 Host PC Tools

  • Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.

📄 Preconditions

  • Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
  • Hardware modifications required for the board to work as endpoint as per the requirements above.
  • pcitest utility should be part Rootfs/NFS for board working as RC.
PCIe connection


📄 Configuration

Linux Driver Configuration for EP

The following config options has to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.

CONFIG_PCI_ENDPOINT=y
CONFIG_PCI_EPF_TEST=y
CONFIG_PCIE_RCAR_EP=y

📄 PCIe EP DTS configuration

By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.

RZ/G2H HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
index 812995939841..9ceb7bbad727 100644
--- a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
@@ -18,3 +18,11 @@
 &sata {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2M HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
index a5ca86196a7b..67802520fb0d 100644
--- a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
@@ -19,3 +19,11 @@
 &pciec1 {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2N HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
index 60d7c8adea02..f7ed164a0e34 100644
--- a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
@@ -19,3 +19,11 @@
 &sata {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2E EK874 board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/cat875.dtsi b/arch/arm64/boot/dts/renesas/cat875.dtsi
index 801ea54b027c..b4edfcaf323e 100644
--- a/arch/arm64/boot/dts/renesas/cat875.dtsi
+++ b/arch/arm64/boot/dts/renesas/cat875.dtsi
@@ -41,6 +41,10 @@
 };
 
 &pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
        status = "okay";
 };

PCIe RC DTS configuration

By default PCIe controller is enabled in root complex mode in the DTS.

PCIe EP Setup

To find the list of endpoint controller devices in the system:

# ls /sys/class/pci_epc/
  fe000000.pcie_ep

To find the list of endpoint function drivers in the system:

# ls /sys/bus/pci-epf/drivers
  pci_epf_test

Using the pci-epf-test function driver

The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported

  • BAR tests
  • Interrupt tests (legacy)
  • Read tests
  • Write tests
  • Copy tests

Creating pci-epf-test device

PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used

# mount -t configfs none /sys/kernel/config
# cd /sys/kernel/config/pci_ep
# mkdir functions/pci_epf_test/func1

The “mkdir functions/pci_epf_test/func1” above creates the pci-epf-test function device.

The PCIe endpoint framework populates the directory with configurable fields.

# ls functions/pci_epf_test/func1
  baseclass_code    function       revid      vendorid
  cache_line_size   interrupt_pin  subclass_code
  deviceid          peripheral     subsys_id
  epc               progif_code    subsys_vendor_id

The driver populates these entries with default values when the device is bound to the driver. The pci-epf-test driver populates vendorid with 0xffff and interrupt_pin with 0x0001

# cat vendorid
  0xffff
# cat interrupt_pin
  0x0001

Creating pci-epf-test device

The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.

Configure Renesas as the vendor.

# echo 0x1912 > functions/pci_epf_test/func1/vendorid

Configure the device ID.

# echo 0x0025 > functions/pci_epf_test/func1/deviceid # For RZ/G2H
# echo 0x0028 > functions/pci_epf_test/func1/deviceid # For RZ/G2M
# echo 0x002b > functions/pci_epf_test/func1/deviceid # For RZ/G2N
# echo 0x002d > functions/pci_epf_test/func1/deviceid # For RZ/G2E

Configure MSI Interrupt

Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them.

# echo 1 > functions/pci_epf_test/func1/msi_interrupts

Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.

Binding pci-epf-test device to a EP controller

In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.

# ln -s functions/pci_epf_test/func1/ controllers/fe000000.pcie_ep/

Starting the EP device

In order for the EP device to be ready to establish the link, the following command should be given.

# echo 1 > controllers/fe000000.pcie_ep/start

Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).

PCIe RC

The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesn’t support hot plug.

Endpoint Test function device

pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.

# insmod pci_endpoint_test.ko

Verification of PCIe EP on RC using lspci

root@salvator-x:~# lspci -vv
00:00.0 PCI bridge: Renesas Technology Corp. Device 002d (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 141
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: None
Memory behind bridge: fe200000-fe2fffff [size=1M]
Prefetchable memory behind bridge: None
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000b4924000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis L[ 140.002782] random: crng init done
[ 140.006278] random: 7 urandom warning(s) missed due to ratelimiting
evel: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Unassigned class [ff00]: Renesas Technology Corp. Device 002d
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 132
Region 0: Memory at fe200200 (64-bit, non-prefetchable) [size=128]
Region 2: Memory at fe200000 (64-bit, non-prefetchable) [size=256]
Region 4: Memory at fe200100 (64-bit, non-prefetchable) [size=256]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: pci-endpoint-test

pcitest

Building pcitest

# cd <kernel-dir>
# make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- -C tools/pci/
# cp pcitest <rootfs>/usr/sbin/

Note: The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.

pcitest options

$ pcitest  -h
usage: pcitest [options]
Options:
        -D <dev>                PCI endpoint test device {default: /dev/pci-endpoint-test.0}
        -b <bar num>            BAR test (bar number between 0..5)
        -m <msi num>            MSI test (msi number between 1..32)
        -x <msix num>           MSI-X test (msix number between 1..2048)
        -i <irq type>           Set IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X)
        -I                      Get current IRQ type configured
        -l                      Legacy IRQ test
        -r                      Read buffer test
        -w                      Write buffer test
        -c                      Copy buffer test
        -s <size>               Size of buffer {default: 100KB}
        -h                      Print this help message

BAR test

$ pcitest -b 0
BAR0:           OKAY
$ pcitest -b 2
BAR2:           OKAY
$ pcitest -b 4
BAR4:           OKAY

💡 Note: BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable

Legacy interrupt test

$ pcitest -i 0
SET IRQ TYPE TO LEGACY:         OKAY
$ pcitest -l
LEGACY IRQ:     OKAY

MSI interrupt test

$ pcitest -i 1
SET IRQ TYPE TO MSI:         OKAY

$ pcitest -m x (Note: value of x can range from 1 to value echoed into msi_interrupts)

Example to test MSI1:
$ pcitest -i 1
SET IRQ TYPE TO MSI:         OKAY
$ pcitest -m 1
MSI1:     OKAY

READ test

$ pcitest -r 1024
READ ( 102400 bytes):           OKAY

WRITE test

$ pcitest -w 1024
WRITE ( 102400 bytes):          OKAY

COPY test

$ pcitest -c 1024
COPY ( 102400 bytes):           OKAY

💡 Note: Before running READ/WRITE/COPY test cases make sure the interrupt mode is either set to Legacy/MSI

💡 Known Issues

  • On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch gfp-dma.patch for upstream kernel. 
  • For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above.