Most Powerful Open Source ERP

How to setup TSN to create a multicast keep alive network that has deterministic jitter

Step by step instructions how to create a deterministic TSN network.
  • Last Update:2024-02-27
  • Version:001
  • Language:en

Agenda

  • Setup a TSN network which can be used for running rPLC coupler in a deterministic way
  • Evaluate and test the network's performance and determinism

Requirements

In order to reproduce the steps needed in this article you will need the following hardware components:

  • Edge server (tsn-shuttle) - any decent multi core system which has at least one NIC which supports following standards: IEEE 802.1AS (Time Synchronization) and IEEE 802.1 QBv (Time Aware Shaper). A good example is any NIC with this chipset Intel Corporation Ethernet Controller I225-V.
  • Coupler (stm32mp1-2) - see how you can acquire one here.
  • Coupler (stm32mp1-2) - see how you can acquire one here.
  • any not managed L2 network switch
  • at least 3 RJ45 LAN cables

Goal

When running a control loop processes with a PLC and their respective physical processes (electrical motors running, moving parts, etc) it is extremely important to know and control the precise timing of execution. Shortly said we wanted to have everywhere a deterministic behavior - i.e. be sure that if a certain action (code) to reach a state takes 1ms then it ALWAYS takes 1ms , no more no less.

In the case of a generic PLC the time when a certain state is reached is usually dependent only on the local controller running the logical program controlling the process loop. In the case of rPLC coupler as part of OSIE project we moved the control logic from the PLC itself and moved it to a more powerful computer which is connected LAN network to the coupler. This of course added yet another component to be considered the network itself as the control commands are send from edge server over network to the coupler itself. Thus we had to have a deterministic network.

At end of this howto one will be able to:

  • create a multicast IPv4 network
  • create a time synchronized network based on IEEE 802.1AS
  • create a time shaped network based on IEEE 802.1 QBv
  • run network tests to measure network determinism and performance
  • use created multicast network for setup of couplers' keep-alive network

Network topology explained

As seen on picture below we have one edge server which in network terms runs as a network router and attached to it a L2 network switch. To the switch we have 2 rPLC couplers connected with RJ45 LAN cables. For the purpose of the how-to we use this network range 192.168.2.0-255 range. Each of the couplers has its own statically assigned IPv4 address. The edge server is able to act as a router / gateway and forward packets to outside network (Internet).

Configure default network configuration

Configure edge server

  1. Set default static address for the NIC to which the L2 network switch is connected to. It will act as a gateway also. For this it is required to edit /etc/network/interfaces file and place following code inside: 
    auto enp6s0
iface enp6s0 inet static
  address 192.168.2.1
  netmask 255.255.255.0
  2. Make sure that tsn-shuttle can act as a router and router packets from stm32mp1-2 and stm32mp1-3 couplers. 
    root@tsn-shuttle:~# sysctl -w net.ipv4.ip_forward=1
  3. Make sure that traffic is allowed on designated interface
  4. root@tsn-shuttle:~# iptables -t nat -A POSTROUTING -o enp6s0 -j MASQUERADE
root@tsn-shuttle:~# iptables -I FORWARD -o enp6s0 -s 192.168.0.0/16 -j ACCEPT
root@tsn-shuttle:~# iptables -I INPUT -s 192.168.0.0/16 -j ACCEPT
# save persistently between reboots
root@tsn-shuttle:~# iptables-save
  5. Restart network for this interface with 
    root@tsn-shuttle:~# ifup enp6s0

Configure coupler

Note: this step needs to be executed for the two couplers. The only difference between them is the IPv4 address which should be as follows:

  • stm32mp1-2 with IPv4 = 192.168.2.2
  • stm32mp1-3 with IPv4 = 192.168.2.3
  1. Set default static address for the NIC to which coupler is connected to the L2 network switch. For this it is required to edit /etc/network/interfaces file and place following code inside: 
    auto eth0
iface eth0 inet static
  address 192.168.2.2
  gateway 192.168.2.1
  netmask 255.255.255.0
  2. Restart network for this interface with 
    root@stm32mp1-2:~# ifup eth0

Make sure that all machines are ping-able

We need to assure that all of the above configurations are working properly. The best way is to use ping

root@tsn-shuttle:~# ping 192.168.2.2
root@tsn-shuttle:~# ping 192.168.2.3
# and from couplers to edge server
root@stm32mp1-2:~# ping 192.168.2.1
root@stm32mp1-3:~# ping 192.168.2.1

Configure time synchronization between all machines

We need to make sure that all machines have their time precisely set. For this we use the following standard IEEE 802.1AS and linuxptp project

Install linuxptp on all machines

Note: even though linuxptp is packaged for almost all OS distributions we will compile from source as follows. It is required to do on all machines!

root@stm32mp1-3:~# wget https://sourceforge.net/projects/linuxptp/files/v3.1/linuxptp-3.1.1.tgz
root@stm32mp1-3:~# tar -xvf linuxptp-3.1.1.tgz
root@stm32mp1-3:~# cd linuxptp-3.1.1
root@stm32mp1-3:~/linuxptp-3.1.1# make
root@stm32mp1-3:~/linuxptp-3.1.1# make install

Start MASTER clock on edge server

Following command will start synchronization process at edge server to all clients on interface enp6s0. Note: this command should be run within a screen!

root@tsn-shuttle:~# ptp4l -2 -H -i enp6s0 -m --socket_priority=7  --tx_timestamp_timeout=40

Start SLAVE clock on couplers

Following command will start synchronization process from edge server. Note that this command should be run within a screen! Also it should be done also for two coupler machines.

root@stm32mp1-2:~# ptp4l -2 -s -i eth0 -m --socket_priority=7  --tx_timestamp_timeout=40

Sync physical to system time

Following command will copy the so called PHY (physical time) as emitted from edge server to the local system time used by all application on the system. Note that this command should be run within a screen! Also it should be done also for all machines as shown below.

root@tsn-shuttle:~# phc2sys -s enp6s0 -O 0 &
root@stm32mp1-2:~# phc2sys -s eth0 -O 0 &
root@stm32mp1-3:~# phc2sys -s eth0 -O 0 &

Configure multicast network

Configure multicast on edge server

root@tsn-shuttle:~# ifconfig enp2s0 multicast
root@tsn-shuttle:~# route -n add -net 224.0.0.0 netmask 240.0.0.0 dev enp6s0

Configure multicast on coupler

Note: these command must be executed on both of the connected couplers.

root@stm32mp1-2:~# ifconfig eth0 multicast
root@stm32mp1-2:~# route -n add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

Test multicast works

Compile tool for sending and receiving multicast messages

Note: you need to do on all machines and this will generate needed commands for you (msend and mreceive).

root@tsn-shuttle:~# git clone https://github.com/troglobit/mtools.git
root@tsn-shuttle:~# cd mtools/
root@tsn-shuttle:~/mtools# make
root@tsn-shuttle:~/mtools# make install

Really test if multicast works

# start sending multicast messages from edge server
root@tsn-shuttle:~# msend

# and run mreceive command on both couplers
root@stm32mp1-2:~# mreceive
root@stm32mp1-3:~# mreceive

Note: if everything works as expected one can see messages emitted from edge server to both of couplers in the terminal.

Configure TSN network shaping

Note: this part of the how-to will adjust network into 4 different classes. It is done just as an example as real configuration and number of classes depends on the desired usage.

Configure shaping using tc

Note: this command will create 4 network classes and assign to each of it equal in time interval. It is only given as an example.

root@tsn-shuttle:~# tc qdisc replace dev enp6s0 handle 100 parent root taprio num_tc 4 \
                            map 3 3 3 3 3 2 1 0 3 3 3 3 3 3 3 3 \
                            queues 1@0 1@1 1@2 1@3 \
                            base-time 1528743495910289987 \
                            sched-entry S 0x01 250000 \
                            sched-entry S 0x02 250000 \
                            sched-entry S 0x04 250000 \
                            sched-entry S 0xf8 250000 \
                            clockid CLOCK_TAI \
                            flags 0x00

Modify network class using iptables

Note: this command will set network class 4 for traffic using TCP protocol to destination port 5201. It is only given as an example.

root@tsn-shuttle:~# iptables -t nat -A PREROUTING -p tcp  --dport 5201 -j CLASSIFY --set-class 0:4

Network performance and testing

In setup above we used TSN shaping to setup a network with 4 different TSN classes. As evaluating a network can differ from one network to another we can only give a very basic example based on settings above (chapter Configure TSN setting network).

For our example we want to make sure that traffic which is generated from client (stm32mp1-2 coupler) to edge server (tsn-shuttle) is using a specific class and doesn't influence any other control PLC traffic. This also will prove that TSN shapping works and that it is deterministic. For further evaluation of the deterministic behavior one ca see following article, chapter "Measurement 1: Beremiz PLC control cycle latency".

Run iperf3 at edge-server (in a server mode)

root@tsn-shuttle:~# iperf3 -s

Run iperf3 at coupler (in a client mode)

root@stm32mp1-2:~# iperf3 -c 192.168.2.1 -t 100

Monitor in real time the traffic control queue statistics

Note following command (see below) below will show the number of bytes which are transferred over one of the 4 traffic class or queues how they are referred to in the command's output. If your setup is proper then two queues will be used as follows:

  • qdisc pfifo 0: parent 100:4 - for traffic comming from iperf3
  • qdisc pfifo 0: parent 100:1 - for traffic generated by PTP4L commands (see chapter "Configure time synchronization between all machines")
root@tsn-shuttle:~# watch -d tc -s qdisc show dev enp2s0
qdisc taprio 100: root refcnt 5 tc 4 map 3 3 3 3 3 2 1 0 3 3 3 3 3 3 3 3
queues offset 0 count 1 offset 1 count 1 offset 2 count 1 offset 3 count 1
clockid TAI	base-time 1528743495910289987 cycle-time 1000000 cycle-time-extension 0
	index 0 cmd S gatemask 0x1 interval 250000
	index 1 cmd S gatemask 0x2 interval 250000
	index 2 cmd S gatemask 0x4 interval 250000
	index 3 cmd S gatemask 0xf8 interval 250000

 Sent 182226698 bytes 2139557 pkt (dropped 0, overlimits 0 requeues 76) 
 backlog 0b 0p requeues 76
qdisc pfifo 0: parent 100:4 limit 1000p
 Sent 175772440 bytes 2039913 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:3 limit 1000p
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:2 limit 1000p
 Sent 52020 bytes 578 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:1 limit 1000p
 Sent 6402238 bytes 99066 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

If for the sake of experiment one want to check if / how trafic shaping works one can adjust  --socket_priority argument to another value (see chapter "Configure time synchronization between all machines").

Conclusions

Your network setup is ready for Beremiz IDE/runtime and rPLC.