EPICS for the Bewildered

The IOC (VME running vxWorks)

First and foremost, you'll need to get an IOC which can talk to your thingamajig (or, more likely, a computer which can talk to the card which talks to your thingamajig). As of EPICS 3.14.3, the computer you can use as your IOC is pretty wide open and is more a function of OS than hardware. We use VME crates (running vxWorks on a Motorola processor) here at the beam-line, but you can just as easily use a Linux box or even a Mac.

• Assembling the VME crate

To get started, you need:

One (1) VME crate model 5099KDL20FK-100 (this model can reboot without resetting the cards on the bus).
One (1) Motorola Controller card.
Most of our controller cards are Motorolla MVME162-532A (MC68040 Processor 32 MHz, 16MB DRAM). Unfortunately, this model has been discontinued. The replacement part is the MVME162PA-344SE, but why use an old processor?

The GSECARS beamline is successfully using the MVME2700-3251A (MPC750 Processor, 233 MHz, 128MB ECC DRAM)
The GM/CA beamline is successfully using the MVME2100 (MPC8240 Processor, 200 MHz, 32MB SDRAM [ECC optional])
A slightly better option is the MVME5100 (MPC750 Processor, 450 MHz, 512 MB ECC SDRAM)
One (1) Transition module (MVME2700).
This is a card which plugs into the back of the VME crate and shares the P2 connector with the controller card. It provides further communication ports and an ethernet connection for communication to the LAN. For the MVME2700, we need the MVME761 transition board.
BEWARE: The MVME761 transition board assembly contains a "paddle" board, a ribbon cable, and then the transition board itself. Connect the "paddle" board to the P2 connector on the VME crate. It is possible to reverse the order of the ribbon cable and paddle board. If you reverse the order, you will CRACK the LSI LOGIC chip on the CONTROLLER CARD !! (Yes, you read that correctly: the MVME2700 controller card will make an audible "pop" and you will release the magic smoke).

Once these are assembled according to Motorola's instructions (TRIPLE CHECK the connections to the transition module if you're using it), you need to connect the serial port (COM1 on the 761 transition board (DB9M) and DEBUG on the MVME5100) to your PC so that you can talk to the controller using something like minicom. For the 761 transition board I just used a stadard female-female DB9 cable. However, the RJ-45 serial connector for the MVME5100 is non-standard and has the following (null modem) pin-outs:

RJ-45	1	2	3	4	5	6	7	8
Signal	DCD	RTS	GND	TxD	RxD	GND	CTS	DTR
DB9	1	8	5	2	3	N/C	7	6

The connection was 9600 8N1.

Once you have successfully connected to the controller, on power-up, you should see this:

For the 2700...
Copyright Motorola Inc. 1988 - 1998, All Rights Reserved                       

PPC1 Debugger/Diagnostics Release Version 4.2 - 11/12/98 RM01                
COLD Start
 
Local Memory Found =08000000 (&134217728)

MPU Clock Speed =233Mhz                                                        

BUS Clock Speed =67Mhz                                                       

WARNING: Keyboard Not Connected                                            

Reset Vector Location  : ROM Bank A                                      
Mezzanine Configuration: Single-MPU                                     
Current 60X-Bus Master : MPU0                                          
Idle MPU(s)            : NONE                                         

System Memory: 128MB, ECC Enabled (ECC-Memory Detected)             
L2Cache:       1024KB, 117Mhz                                      
 
PPC1-Bug>

For the 5100...
Copyright Motorola Inc. 1988 - 2002, All Rights Reserved                       

PPC6 Debugger/Diagnostics Release Version 2.2 - 04/16/02 RM03                
COLD Start
 
Local Memory Found =20000000 (&536870912)

MPU Clock Speed =450Mhz                                                        

BUS Clock Speed =100Mhz                                                       

Reset Vector Location  : ROM Bank B                                      
Mezzanine Configuration: Single-MPU                                     
Current 60X-Bus Master : MPU0                                          
Idle MPU(s)            : NONE                                         

L2Cache:       1024KB, 180Mhz                                      
System Memory: 512MB, ECC Enabled (ECC-Memory Detected)             
 
PPC6-Bug>

The instructions for both the PPC-Bug and PPC-Diag commands are located on the Motorola Computer Group's Web Page.

• Installing vxWorks

At the moment, the controller card is just running a simple debugger. It's now time to install an operating system. Here at the beam-line, we seem to have a preference for vxWorks because we're government funded.

First and foremost, you will need two files: The vxWorks boot image (to be loaded into the controller card's flash memory) and the vxWorks kernel image. Both of these binary files are created under the target/config/<board> directory of the Tornado distribution.

(It may be necessary to first build the cross-compiler if you don't have the vxWorks binaries in hand. I already had the binaries, so I didn't have to do this, but you might want to skip ahead and do that before coming back here).

= Installing the vxWorks Boot Image

Once the bootrom and kernel image have been made, we have to get the vxWorks boot image onto a machine connected to the LAN. This will involve setting up a TFTP server somewhere on your network:

Install the tftp-server rpm on the server machine.
Create a world-readable directory for tftp (say, /var/tftp)
In the file "/etc/xinetd.d/tftp" :
1. Change "disable" to "no".
2. Add an "only_from =" line to control access. (man xinetd.conf)
3. Change the "server_args" -s argument to your tftp directory
Make sure tftp is chkconfig'ed on.
/etc/init.d/xinetd restart

Now follow the instructions for installing the VxWorks boot image for your controller card. Below, I mention some caveats for various boards.

The MVME2700 and MVME5100 have of soldered flash EPROM as well as 1MB of socketed Flash EPROM. [The MVME2700 has 4 MB of soldered flash and the MVME5100 uses four 4 MB soldered AMD-D323GT90V1 chips for bank "A" (for a total of 16 MB) and two 512 KB 32-pin AM29LV040B chips for bank "B"]

The goal is to load the vxWorks boot image into the soldered EPROM in bank "A" (faster CPU access) while operating under the PPCx-Bug image installed in the socketed flash in bank "B". Then we boot into the new OS on the soldered flash.

A board jumper determines which EPROM to boot from. For the MVME2700, it's J9 "(near the board's bottom handle)". For the MVME5100, it's J7 "(next to the screw holding down the heatsink)". Remove power from the crate. Set this jumper so that it connects pins 2 and 3. This selects the socketed EPROM bank "B." We now attempt to put the vxWorks boot image into the soldered flash in bank "A."

Connect the ethernet cable to the controller (or 761 transition board), and power up the system. You should see the following line in the boot sequence:

Reset Vector Location  : ROM Bank B

Once you get to the "PPCx-Bug>" prompt, the first order of business is to set the system clock:

  set MMDDYYhhmm

I typically set GMT. You can query the time using the "time" command.

Now we need to tell the computer from where to download the image. This is done using the program "niot." The "Client IP Address" will be the IP address of the board and the "Server IP Address" is the IP address of the machine which has the image.

A response of "." to any question exits the program.

   PCx-Bug> niot
   Controller LUN =00?
   Device LUN     =00?
   Node Control Memory Address =xxxxxxxx?
   Client IP Address      =0.0.0.0? 164.54.204.xxx
   Server IP Address      =0.0.0.0? 164.54.204.yyy
   Subnet IP Address Mask =255.255.255.0? 
   Broadcast IP Address   =255.255.255.255? 164.54.204.255
   Gateway IP Address     =0.0.0.0? 164.54.204.___
   Boot File Name ("NULL" for None)     =? .

   Update Non-Volatile RAM (Y/N)? y
   PPCx-Bug>

Let's test that setting with the "nping" command:

PPCx-Bug>nping 0 0 164.54.204.xxx 164.54.204.yyy 1

Which asks the source (164.54.204.x) to ping the destination (164.54.204.y) one time (the last parameter is in hexadecimal). You should get the response

Source IP Address      = 164.54.204.xxx
Destination IP Address = 164.54.204.yyy
Number of Packets Transmitted =1, Packets Lost =0, Packet Size =128

N.B.: If there was recently another device on the server IP address (such as a board you just programmed), it may take some time for the ARP database to refresh on the gateway (on the order of 15 minutes). This can be remedied if you have root priveleges on the tftp server. First, find the controller card's Ethernet address by typing "cnfg". Then, on the server (as root), execute:

server# arp -s <client IP> <client ethernet address>

Don't forget to add the colons to the ethernet address (cnfg will just show a string of 12 hex characters).

Once this works, the image is downloaded (via tftp) into memory via the "niop" command. "The file name below is set to the location of the boot image in [the server's] TFTP area:"

PPCx-Bug> niop
   Controller LUN =00?
   Device LUN     =00?
   Get/Put        =G?
   File Name      =? vxworks/bootrom.mvXXXX
   Memory Address =00004000?
   Length         =00000000?
   Byte Offset    =00000000?
...
   PPCx-Bug>

Notice that the image begins at address 4000.

"[Finally], the "pflash" command is used to put [the boot image] into the FLASH chips soldered onto the mezzanine board." The command is defined as

pflash <Source Base Address>:<Source End Address> <Destination Address>

For the MVME2700 the destination address is FF000100. However, for the MVME5100 the destination address depends on the size of the soldered flash.

64 KB of processor memory (beginning at FEF80000) is devoted to the system memory controller (SMC) registers. The "ROM A" base address is loated in the 12 bits beginning at FEF80050. Type:

PPC6-Bug> md fef80050
FEF80050  F40C0006 00000000 FF800006 00000000  ................
FEF80060  00000000 00000000 00000000 00000000  ................

So the ROM A flash addressing begins at F4000000 (the "C" in the lowest four bits of the second byte means that rom_a_64=1 (64 bits wide) and the size of ROM A is "100" which is code for 16 MB). We begin loading the flash memory 256 bits (32 bytes) in. In this case, that would be at address F4000100. So now type:

   PPCx-Bug> pflash 4000:FFF00 <DEST>

where <DEST> = FF000100 for an MVME2700 or F4000100 for the MVME5100 described above.

   Program FLASH Memory (Y/N)? y
...
   FLASH Memory Programming Complete                  
   PPCx-Bug>

At this point, we can boot from the vxWorks image. Power down the crate and replace the ROM select jumper (J9 or J7) across pins 1 and 2. When power is re-applied, the vxWorks boot countdown should start. "You'll need to stop this before it finishes to set the vxWorks boot parameters...."

End of thievery, Andrew ;-)

= Configuring the Network (rsh issues)

At this point we need to delineate the host linux box. We have decided to use "rsh" which, as you know, is not terribly secure. The following circumstances seem to make RSH "ok" for now:

We set the host:/etc/xinetd.d/rsh file to accept connections only inside the subnet (man xinetd.conf)
No sensitive information will be transferred
A .rhosts file in the IOC's home directory will keep passwords from being sent cleartext
When testing rsh from another computer, make sure iptables is not blocking the return connection or you will see
poll: protocol failure in circuit setup

Set up rsh and place the current vxWorks binary under host:/home/<username>/<subdir>/

= Configuring the vxWorks Boot Program

Now that the vxWorks boot executable has been loaded into flash memory, we need to configure it to download the actual vxWorks OS. This is done by halting the auto-boot and typing "c" at the "[VxWorks Boot]:" prompt. The following is a description of our configuration:

[VxWorks Boot]: c

'.' = clear field;  '-' = go to previous field;  ^D = quit

 boot device          : dc0 / fei0
 processor number     : 0
 host name            : iochost
 file name            : /home/username/crate3/mv2700-asd4
 inet on ethernet (e) : 164.54.204.xx:aabbccdd
 inet on backplane (b):
 host inet (h)        : 164.54.204.yy
 gateway inet (g)     :
 user (u)             : username
 ftp password (pw) (blank = use rsh):
 flags (f)            : 0x0
 target name (tn)     : crate3
 startup script (s)   : /home/username/crate3/st.cmd 
 other (o)            :

boot device
This tells the processor from where to get its instructions. For the MVME2700, "dc0" is the ethernet device. For the MVME5100, it's "fei0"
processor number
On a single processor board, this should be "0"
host name
The name of the host holding the startup files.
file name
The full pathname to the vxWorks kernel image.
inet on ethernet (e)
The VME's assigned IP address. The subnet mask is in hex following the colon (ours is ffffffc0).
inet on backplane (b)
We don't use this....
host inet (h)
The hosts IP address.
gateway inet (g)
The IP address of the gateway (not necessary if not leaving the subnet).
ftp password (pw)
We currently use "rsh." (this is somewhat insecure and needs to be addressed by Wind River Systems).
flags (f)
We don't use this....
target name (tn)
The name of the VME crate.
startup script (s)
The ASCII file which the VME crate will execute upon loading its kernel image. At first, this file can be blank or hold some printf statements.

Once this is set up, type "@" and hope the VME crate can load the files it needs from the host. If it works, you should get a vxWorks prompt and you can start playing with the vxWorks OS.

Useful vxWorks Commands
version	Prints out the vxWorks version information and boot line
i	Equivalent to linux "ps"
<	Imports a text file of commands (e.g. < my_script )
ld <	Imports compiled code (e.g, ld < iocCore )
d 0xaddress	Dump memory starting at hex location address In some cases, "d" will fail. Here is one reason from the documention of a card: "NOTE: the VSAM data must be accessed as A24/D32. If you attempt to access the VSAM data from the vxWorks shell, using the "d" command, you will get an access violation. This is because "d" accesses the data 1-byte at a time."
vxMemProbe(0xaddress, RW, BYTES, LVAR)	Will probe an address and trap bus errors (exit -1 if bus error) Use: ioc> x = 1 ioc> vxMemProbe (0xaddress, 0x0, 4, &x) ioc> x value = 0xVALUE RW is 0x0 for read and 0x1 for write LVAR is the variable to read into or write from ADDRESS
sysBusToLocalAdrs(0xAMC, 0xaddress, &VARIABLE)	Will tell you where a VME bus address is mapped to in VxWorks memory. Use: ioc> x = 1 ioc> sysBusToLocalAdrs(0x39, 0xBUS_ADDRESS, &x) ioc> x value = 0xLOCAL_ADDRESS The Address Modifier Codes are found in vme.h 0x09 A32 "user privilege" Data 0x0a A32 "user privilege" program instruction fetch 0x0b A32 "user privilege" Block Transfer 0x0d A32 "supervisor priv" Data 0x0e A32 "supervisor priv" program instruction fetch 0x0f A32 "supervisor priv" Block transfer 0x29 A16 "user privilege" I/O 0x2d A16 "supervisor priv" I/O 0x39 A24 "user privilege" Data 0x3a A24 "user privilege" program instruction fetch 0x3b A24 "user privilege" Block transfer 0x3d A24 "supervisor priv" Data 0x3e A24 "supervisor priv" program instruction fetch 0x3f A24 "supervisor priv" Block transfer = AMCs of choice

Also useful: Chethan Parameswariah's VxWorks Command Cheat Sheet.

For example:
We have a Struck scaler at address $a0 00 00 00. The first four bytes of address space are its control register. Bit "0" lights the lamp if set. We can light the user lamp thusly:

First, because we must read and write four bytes at a time, let's find out what's there before we mess with it:

  ioc> x = 0 ; vxMemProbe(0xa0000000, 0x0, 4, &x) ; x
  x = 0x6d5e28: value = 0 = 0x0
  value = 0 = 0x0
  x = 0x6d5e28: value = 9380232 = 0x8f2188

So we know that the four bytes are 00 8f 21 88. To light the user lamp, we set bit zero: y = 0x008f2189 ; vxMemProbe(0xa0000000, 0x1, 4, &y) Turning off the user lamp requires clearing the low bit: y = 0x008f2188 ; vxMemProbe(0xa0000000, 0x1, 4, &y) And we can read the module identification at an offset of 0x4: x = 0 ; vxMemProbe(0xa0000004, 0x0, 4, &x) ; x N.B. This is a case where "d 0xa0000000" will fail.

(Installing the EPICS Client) <-BACK | NEXT-> (Installing the EPICS Server)