/alps/pcitool

Memory Addressing

=================

 There is 3 types of addresses: virtual, physical, and bus. For DMA a bus

 address is used. However, on x86 physical and  bus addresses are the same (on

 other architectures it is not guaranteed). Anyway, this assumption is still

 used by xdma driver, it uses phiscal address for DMA access. I have ported

 in the same way. Now, we need to provide additionaly bus-addresses in kmem

 abstraction and use it in NWL DMA implementation.

DMA Access Synchronization

==========================

 - At driver level, few types of buffers are supported:

    * SIMPLE - non-reusable buffers, the use infomation can be used for cleanup

    after crashed applications.

    * EXCLUSIVE - reusable buffers which can be mmaped by a single appliction

    only. There is two modes of these buffers:

	+ Buffers in a STANDARD mode are created for a single DMA operation and

	if such buffer is detected while trying to reuse, the last operation

	has failed and reset is needed.

	+ Buffers in a PERSISTENT mode are preserved between invocations of

	control application and cleaned up only after the PERSISTENT flag is 

	removed

    * SHARED - reusable buffers shared by multiple processes. Not really 

    needed at the moment.

    KMEM_FLAG_HW - indicates that buffer can be used by hardware, acually this

    means that DMA will be enabled afterwards. The driver is not able to check

    if it really was enable and therefore will block any attempt to release 

    buffer until KMEM_HW_FLAG is passed to kmem_free routine as well. The later

    should only called with KMEM_HW_FLAG after the DMA engine is stopped. Then,

    the driver can be realesd by kmem_free if ref count reaches 0.

    KMEM_FLAG_EXCLUSIVE - prevents multiple processes mmaping the buffer 

    simultaneously. This is used to prevent multiple processes use the same

    DMA engine at the same time. When passed to kmem_free, allows to clean

    buffers with lost clients even for shared buffers.

    KMEM_FLAG_REUSE - requires reuse of existing buffer. If reusable buffer is 

    found (non-reusable buffers, i.e. allocated without KMEM_FLAG_REUSE are

    ignored), it is returned instead of allocation. Three types of usage 

    counters are used. At moment of allocation, the HW reference is set if 

    neccessary. The usage counter is increased by kmem_alloc function and

    decreased by kmem_free. Finally, the reference is obtained at returned

    during mmap/munmap. So, on kmem_free, we do not clean

	a) buffers with reference count above zero or hardware reference set.

	REUSE flag should be supplied, overwise the error is returned

	b) PERSISTENT buffer. REUSE flash should be supplied, overwise the 

	error is returned

	c) non-exclusive buffers with usage counter above zero (For exclusive

	buffer the value of usage counter above zero just means that application

        have failed without cleaning buffers first. There is no easy way to 

        detect that for shared buffers, so it is left as manual operation in

        this case)

        d) any buffer if KMEM_FLAG_REUSE was provided to function

    During module unload, only buffers with references can prevent cleanup. In

    this case the only possiblity to free the driver is to call kmem_free 

    passing FORCE flags.

    KMEM_FLAG_PERSISTENT - if passed to allocation routine, changes mode of 

    buffer to PERSISTENT, if passed to free routine, vice-versa changes mode

    of buffer to NORMAL. Basically, if we call 'pci --dma-start' this flag

    should be passed to alloc and if we call 'pci --dma-stop' it should be

    passed to free. In other case, the flag should not be present.

    If application crashed, the munmap while be still called cleaning software

    references. However, the hardware reference will stay since it is not clear

    if hardware channel was closed or not. To lift hardware reference, the 

    application can be re-executed (or dma_stop called, for instance).

    * If there is no hardware reference, the buffers will be reused by next 

    call to application and for EXCLUSIVE buffer cleaned at the end. For SHARED

    buffers they will be cleaned during module cleanup only (no active 

    references).

    * The buffer will be reused by next call which can result in wrong behaviour

    if buffer left in incoherent stage. This should be handled on upper level.

 - At pcilib/kmem level synchronization of multiple buffers is performed

    * The HW reference and following modes should be consistent between member 

    parts: REUSABLE, PERSISTENT, EXCLUSIVE (only HW reference and PERSISTENT 

    mode should be checked, others are handled on dirver level)

    * It is fine if only part of buffers are reused and others are newly 

    allocated. However, on higher level this can be checked and resulting

    in failure.

    Treatment of inconsistencies:

     * Buffers are in PRESISTENT mode, but newly allocated, OK

     * Buffers are reused, but are not in PERSISTENT mode (for EXCLUSIVE buffers

     this means that application has crashed during the last execution), OK

     * Some of buffers are reused (not just REUSABLE, but actually reused), 

     others - not, OK until 

        a) either PERSISTENT flag is set or reused buffers are non-PERSISTENT

	b) either HW flag is set or reused buffers does not hold HW reference

     * PERSISTENT mode inconsistency, FAIL (even if we are going to set 

     PERSISTENT mode anyway)

     * HW reference inconsistency, FAIL (even if we are going to set 

     HW flag anyway)

    On allocation error at some of the buffer, call clean routine and

     * Preserve PERSISTENT mode and HW reference if buffers held them before

     unsuccessful kmem initialization. Until the last failed block, the blocks

     of kmem should be consistent. The HW/PERSISTENT flags should be removed

     if all reused blocks were in HW/PERSISTENT mode. The last block needs

     special treatment. The flags may be removed for the block if it was

     HW/PERSISTENT state (and others not).

     * Remove REUSE flag, we want to clean if allowed by current buffer status

     * EXCLUSIVE flag is not important for kmem_free routine.

 - At DMA level

    There is 4 components of DMA access:

    * DMA engine enabled/disabled

    * DMA engine IRQs enabled/disabled - always enabled at startup

    * Memory buffers

    * Ring start/stop pointers

    To prevent multiple processes accessing DMA engine in parallel, the first

    action is buffer initialization which will fail if buffers already used

	* Always with REUSE, EXCLUSIVE, and HW flags 

	* Optionally with PERSISTENT flag (if DMA_PERSISTENT flag is set)

    If another DMA app is running, the buffer allocation will fail (no dma_stop 

    is executed in this case) 

    Depending on PRESERVE flag, kmem_free will be called with REUSE flag 

    keeping buffer in memory (this is redundant since HW flag is enough) or HW

    flag indicating that DMA engine is stopped and buffer could be cleaned.

    PERSISTENT flag is defined by DMA_PERSISTENT flag passed to stop routine.

    PRESERVE flag is enforced if DMA_PERSISTENT is not passed to dma_stop

    routine and either it:

	a) Explicitely set by DMA_PERMANENT flag passed to dma_start 

	function 

	b) Implicitely set if DMA engine is already enabled during dma_start, 

	all buffers are reused, and are in persistent mode.

    If PRESERVE flag is on, the engine will not be stopped at the end of

    execution (and buffers will stay because of HW flag).

    If buffers are reused and are already in PERSISTENT mode, DMA engine was on 

    before dma_start (PRESERVE flag is ignored, because it can be enforced), 

    ring pointers are calculated from LAST_BD and states of ring elements.

    If previous application crashed (i.e. buffers may be corrupted). Two

    cases are possible:

    * If during the call buffers were in non-PERSISTENT mode, it can be 

    easily detected - buffers are reused, but are not in PERSISTENT mode 

    (or at least was not before we set them to). In this case we just 

    reinitialize all buffers.

    * If during the call buffers were in PERSISTENT mode, it is up to 

    user to check their consistency and restart DMA engine.]

    IRQs are enabled and disabled at each call

DMA Reads

=========

standard: 		default reading mode, reads a single full packet

multipacket:		reads all available packets

waiting multipacket:	reads all available packets, after finishing the

			last one waiting if new data arrives

exact read:		read exactly specified number of bytes (should be

			only supported if it is multiple of packets, otherwise

			error should be returned)

ignore packets:		autoterminate each buffer, depends on engine 

			configuration

 To handle differnt cases, the value returned by callback function instructs

the DMA library how long to wait for the next data to appear before timing 

out. The following variants are possible:

terminate:		just bail out

check:			no timeout, just check if there is data, otherwise 

			terminate

timeout:		standard DMA timeout, normaly used while receiving

			fragments of packet: in this case it is expected 

			that device has already prepared data and only

			the performance of DMA engine limits transfer speed

wait:			wait until the data is prepared by the device, this

			timeout is specified as argument to the dma_stream

			function (standard DMA timeout is used by default)

			first |  new_pkt  | bufer 

			--------------------------	

standard		wait  | term      | timeout  

multiple packets	wait  | check	  | timeout 	- DMA_READ_FLAG_MULTIPACKET 	

waiting multipacket	wait  | wait      | timeout 	- DMA_READ_FLAG_WAIT

exact			wait  | wait/term | timeout	- limited by size parameter

ignore packets		wait  | wait/check| wait/check 	- just autoterminated

Shall we do a special handling in case of overflow?

Buffering

=========

 The DMA addresses are limited to 32 bits (~4GB for everything). This means we 

 can't really use DMA pages are sole buffers. Therefore, a second thread, with

 a realtime scheduling policy if possible, will be spawned and will copy the 

 data from the DMA pages into the allocated buffers. On expiration of duration

 or number of events set by autostop call, this thread will be stopped but 

 processing in streaming mode will continue until all copyied data is passed 

 to the callbacks.

 To avoid stalls, the IPECamera requires data to be read continuously read out.

 For this reason, there is no locks in the readout thread. It will simplify

 overwrite the old frames if data is not copied out timely. To handle this case

 after getting the data and processing it, the calling application should use

 return_data function and check return code. This function may return error

 indicating that the data was overwritten meanwhile. Hence, the data is 

 corrupted and shoud be droped by the application. The copy_data function

 performs this check and user application can be sure it get coherent data

 in this case.

 There is a way to avoid this problem. For raw data, the rawdata callback

 can be requested. This callback blocks execution of readout thread and 

 data may be treated safely by calling application. However, this may 

 cause problems to electronics. Therefore, only memcpy should be performed

 on the data normally. 

 The reconstructed data, however, may be safely accessed. As described above,

 the raw data will be continuously overwritten by the reader thread. However,

 reconstructed data, upon the get_data call, will be protected by the mutex.

Register Access Synchronization

===============================

 We need to serialize access to the registers by the different running 

 applications and handle case when registers are accessed indirectly by

 writting PCI BARs (DMA implementations, for instance).

 - Module-assisted locking:

 * During initialization the locking context is created (which is basicaly

 a kmem_handle of type LOCK_PAGE. 

 * This locking context is passed to the kernel module along with lock type 

 (LOCK_BANK) and lock item (BANK ADDRESS). If lock context is already owns

 lock on the specified bank, just reference number is increased, otherwise

 we are trying to obtain new lock.

 * Kernel module just iterates over all registered lock pages and checks if

 any holds the specified lock. if not, the lock is obtained and registered

 in the our lock page.

 * This allows to share access between multiple threads of single application

 (by using the same lock page) or protect (by using own lock pages by each of

 the threads)

 * Either on application cleanup or if application crashed, the memory mapping

 of lock page is removed and, hence, locks are freed.

 - Multiple-ways of accessing registers

 Because of reference counting, we can successfully obtain locks multiple 

 times if necessary. The following locks are protecting register access:

  a) Global register_read/write lock bank before executing implementation

  b) DMA bank is locked by global DMA functions. So we can access the 

  registers using plain PCI bar read/write.

  c) Sequence of register operations can be protected with pcilib_lock_bank

  function

 Reading raw register space or PCI bank is not locked.

  * Ok. We can detect banks which will be affected by PCI read/write and 

  lock them. But shall we do it?

Register/DMA Configuration

==========================

 - XML description of registers

 - Formal XML-based (or non XML-based) language for DMA implementation. 

   a) Writting/Reading register values

   b) Wait until <register1>=<value> on <register2>=<value> report error

   c) ... ?

IRQ Handling

============

 IRQ types: DMA IRQ, Event IRQ, other types

 IRQ hardware source: To allow purely user-space implementation, as general

 rule, only a  single (standard) source should be used.

 IRQ source: The dma/event engines, however, may detail this hardware source

 and produce real IRQ source basing on the values of registers. For example, 

 for DMA IRQs the source may present engine number and for Event IRQs the 

 source may present event type.

 Only types can be enabled or disabled. The sources are enabled/disabled

 by enabling/disabling correspondent DMA engines or Event types. The expected

 workflow is following:

 * We enabling IRQs in user-space (normally setting some registers). Normally,

 just an Event IRQs, the DMA if necessary will be managed by DMA engine itself.

 * We waiting for standard IRQ from hardware (driver)

 * In the user space, we are checking registers to find out the real source

 of IRQ (driver reports us just hardware source), generating appropriate 

 events, and acknowledge IRQ. This is dependent on implementation and should 

 be managed inside event API.

 I.e. the driver implements just two methods pcilib_wait_irq(hw_source), 

 pcilib_clear_irq(hw_source). Only a few hardware IRQ sources are defined.

 In most cirstumances, the IRQ_SOURCE_DEFAULT is used. 

 The DMA engine may provide 3 additional methods, to enable, disable,

 and acknowledge IRQ.

 ... To be decided in details upon the need...

Updating Firmware

=================

 - JTag should be connected to USB connector on the board (next to Ethernet)

 - The computer should be tourned off and on before programming

 - The environment variable should be loaded

    . /home/uros/.bashrc

 - The application is called 'impact'

    No project is needed, cancel initial proposals (No/Cancel)

    Double-click on "Boundary Scan"

    Right click in the right window and select "Init Chain"

    We don't want to select bit file now (Yes and, then, click Cancel)

    Right click on second (right) item and choose "Assign new CF file"

    Select a bit file. Answer No, we don't want to attach SPI to SPI Prom

    Select xv6vlx240t and program it

 - Shutdown and start computer

 Firmware are in

    v.2: /home/uros/Repo/UFO2_last_good_version_UFO2.bit

    v.3: /home/uros/Repo/UFO3 

	Step5 - best working revision

	Step6 - last revision