D22 emergency procedures

Information about D22 can be found on the D22 yellow page in ILL's WWW server.

See also the D22 documentation and the D22 Short Manual.

• 1. Partially missing detector image
• 2. Power failure on supervision PC
• 3. Initialization of selector and detector cooling unit
• 4. Selector restart
• 5. Selector supervision program restart
• 6. Initialization of vacuum pumps
• 7. Collimation time-out
• 8. Power failure on process control cabinet
• 9. Acquisition error
• 10. Low flux
• 11. Restarting MAD
• 12. Rebooting the acquisition workstation d22 (Dec alpha)
• 13. Problems with detector high voltage
• 13.1. Description of the system
• 13.1.1. The HV filter
• 13.1.2. The HV controller
• 13.2. Emergency procedures
• 13.2.1. Problem: the HV supply indicates that the detector is at 0 V without a reason
• 13.2.2. Problem: the high voltage is 0 on the detector, but I'm not allowed to move the detector.
• 13.2.3. Problem: I give a movement command, but the detector stays at 1515 V.
• 13.2.4. The controller is not working
• 13.2.5. The discharge relay is stuck
• 13.2.6. Partially manual mode
• 13.2.7. Live dangerously

In particular after a restart of MAD, parts of the detector (vertical stripes) may not be counting.

In this case, go to the detector acquisition cabin, open the door of the electronics cabinet (after pushing on the buton below the black handle) and push the four reset buttons of the crates where the 256 brown signal cable arrive.

The reset buttons (marked RST) are found on crates that are each positioned in the middle of four crates with lemo connectors.

• C:\> chkdsk /f
• C:\> win

Choose "Gestionnaire de fichiers"

In there, choose "InTouch", "View"

It may be necessary to initialize also the cooling system and the vacuum pumps (see below).

• Select option COOLING in supervision program
• Switch cooling unit off (in page COOLING)
• Reset alarm on control unit (outside cabin): "ACQUIT DEFAUT"
• Switch cooling unit on

The selector normally runs under MAD; if it stops due to an emergency event shown by the red EMERGENCY field in the selector program window on the supervision PC, you should first check whether the emergency is real or a fault of the security system. In the first case, have the fault repaired and reset D22's process controller (ACQUIT DEFAUT button on the central cabinet). In the latter case, just reset.

For restarting the selector after an emergency eveny, it is normally sufficient to push the RESTART button on the selector power supply. This is situated in the lower part of the rightmost cabinet of D22's process controller. If after 5 attempts to restart there has been no success, another RESET is necessary.

If the selector program does not react to any attempt to modify the speed, check whether the selector power supply is on COMPUTER CONTROL. If this is the case, think of restarting the selector program (see below).

The selector supervision program ("Anatole") can be found on the supervision PC in the "Gestionnaire de programmes" in the InTouch directory.

The password is "ILL".

Anatole starts in local mode. The program requires a start value: Click on the CONTROL button and enter an allowed speed value. When this value is reached and stable, you may switch to VAX control by choosing the corresponding menu item ("controle_vax") of the selector program window. If the values imposed by the instrument computer are not read by the selector program although it is in VAX mode, it may also be necessary to restart the communication program MODBUS. Check whether the communication is active: On top of the control PC (under the table), there is a box (westermo) where the RD and TD indicators should blink. If this is not the case, quit MODBUS and start it again from the "Gestionnaire de programmes".

(Quit a window program by clicking its icon and choosing "fermeture" from the menu. It might be necessary to shrink other windows to see the application's icon).

• Select option VACUUM in supervision program
• Switch off "Vax control" in "InTouch" with the MAD command PCCONTRO (PC is sufficient) if necessary
• Type password "mv" in page VACUUM (switch to manual)
• Type password "fmv" in page VACUUM to get control buttons
• Make sure all pumps and valves are correctly set
• (Type password "fvide" to stop access to control buttons)
• Click on right arrow (-->) on bottom right to get to selector vacuum
• Type password "mvs" (manu vide selecteur) for manual choice of pumps etc.
• For changing turbo pump: Reset, type turbo1<cr> or turbo2 <cr> </cr></cr>

Vax control is switched on again automatically by issuing a MAD command addressing the PC, e.g. SEL.

Especially after a shut-down, the collimation change may fail because the positioning pistons have got stuck. If the collimation times out before indicating a correct position, another movement of the collimation is not possible (Message "Collimation is already running").

In this case, wait for the collimation movement to stop,

• Go to the collimation menu in "InTouch"
• Click "Password"
• Type mco
• Click "Password"
• Type fmco

Then type COLL on the MAD terminal and check the current collimation value. After that, a new COLL command should work.

It might be necessary to push the "ACQUIT DEFAUT" button on the control cabinet in the guide hall.

In manual mode, the "START" button (InTouch collimation screen) can be used for setting the collimation.

In case of a power failure of the process control cabinet, rearm the main switch "ARMOIRE CONTROLE COMMANDE", bottom right in the electric cabinet "COFFRET CAR D22" in the hall ILL22, next to the door of the D22 cabin, on the right hand side seen from the hall.

If such a power failure happens, also the pumping and cooling systems and the selector stop (see above).

In the case of an error "Cannot initialize acquisition (check VME)", check whether the red status light of the acquisition card (ILL-TOF), slot 16 in top VME rack, is "half lit". If it isn't

• Exit from MAD (command QUIT)
• Push the reset swich on top of the acquisition card upwards
• Push the reset button on top of the SYSCON card in slot 1 of the VME crate
• Wait for the indicator number on top of the ELTEC card (slot 3) to become "6"
• Start MAD again (type mad [cr] as /users/d22).

Main reasons for a lower flux than expected are

• Beam closed
• Attenuator in place
• Selector failure
• Closed entrance aperture (it is controlled from the module on the left side of control cabinet, third rack from top on the left)

If MAD does not respond to commands, you need to restart all MAD processes.

First, all remaining MAD processes need to be killed (use an Xterm window on the acquisition workstation d22a other than the one running MAD):

     /users/mad/MADKILL/madkill [cr]

Second, you need to restart MAD (in the window where it was running previously or a new one):

     type mad [cr] (as /users/d22)

If none of the windows on the acquisition workstation respond to commands, you may try to switch sessions. Normally you are using the "MAD" session. Try to switch to another session by clicking on "TWO" etc. in the toolbar at the bottom of the workstation screen and go back to the "MAD" session.

If nothing helps, reboot the acquisition workstation:

• Push the ON/OFF button in the upper left corner of the CPU box (marked SI 692 on the right side,top)) behind the door to switch off.
• Push the ON/OFF button again to switch on.
• Type b [carriage return] to start booting (if this doesn't happen automatically)
• Wait until the login window appears on the screen. This takes several minutes.
• Login as d22, password d22d22
• Start an Xterm from the the toolbar at the bottom of the screen (use the arrow on top of the fourth icon of the toolbar)
• Start MAD by typing "mad" followed by a carriage return in the newly created Xterm window
• Create other Xterm sessions as you need them
• Create an Xterm session on the right-hand montor using the command xtermright (on the main monitor)
• Move the cursor to the right monitor and start the processes
  • msp (MAD spy) (use the auto button, refresh 1 sec)
  • visu (2D supervision program) (use the auto button, refresh 5 sec)
  • nsw (printer switching program) (use the PRINTER button, enable online printer)
• You may iconize the unused windows (msp control, nsw, Xterm) by by clicking the dot button on the right side of the window bars.

(Version 2, March 2005, contributed by Patrick van Esch)

The system consists of two units: one is the High voltage (HV) controller and the other is the HV filter. The HV controller (pdf) is in the grey metal box next to the HV supply in the electronics bay in the cabin. The HV filter is mounted inside the cone behind the detector, in the grey metal box on the left side (the upper box ; the lower box is the low voltage distribution box).

Both units are interconnected with 3 lemo cables (of which 2 are "double cables", with two successive cable numbers), named "charge rapide", "decharge" and "mesure". The HV cable number 1 goes directly from the HV power supply onto the HV filter. HV cable 2 is a spare cable.

The HV filter box applies the HV to the detector and can also discharge the detector. There are two HV relays that do this. It also measures the HV on the detector through a resistive divider (divides by 1000). The symbolic scheme is the following:

                      +----(relay 1)----+
                      |                 | 
 (HV cable 1) --------+---(RC filter)---+---+----------+---(HV out 1 and 2) 
                                        |              | 
                                    (relay 2)  (voltage divider)--- "mesure"
                                        |              |
                                       gnd            gnd

Relay 1 is a passively OPEN relay, which closes (makes contact) when we put a voltage on "charge rapide".

Relay 2 is a passively CLOSED relay, which shortens the HV output to ground. When we put a voltage on "decharge", the relay opens (breaks contact with ground) and hence permits a voltage to be present on the detector. This has been choosen for security: if ever the relay cable is cut or disconnected, the detector will discharge (most secure situation).

If relay 2 is not activated (we 'discharge' the detector) and relay 1 is activated (currently this is always the case) then we will also shorten the HV supply ; we want to avoid this situation (as it is thought to be a cause of a sticking relay 2 due to the sudden discharge of the capacity of the HV cable through relay 2). When the detector is discharged, relay 1 will be open. THIS HAS THE EFFECT THAT 1515V WILL REMAIN VISIBLE ON THE HV SUPPLY EVEN IF THE DETECTOR IS DISCHARGED. There is no visual indication anymore of the voltage on the detector.

The voltage divider divides the voltage measured on the HV out by 1000, and this voltage is present on "mesure".

The normal working situation is the following: relay 1 is open (not powered), and relay 2 is open (is powered).  The detector is then under HV, through the filter.
The "discharge" situation is the following: relay 1 is open (not powered), and relay 2 is closed (is not powered).  The detector is then discharged.
Only during a short time, in the "fast charge" mode, relay 1 is closed (powered) and relay 2 is open (powered).  The detector is connected directly (during about 6 seconds) to the HV power supply, and charges up quickly.

The HV controller (pdf) is based upon an industrial process controller. It is powered through two 12 V adapters in the power outlet on the back of the electronics bay for functioning. It has ADC inputs and it has relays as outputs.

The configuration is the following:

Input 5: HV "mesure" signal (on an ADC input)
Input 6: logical signal (movement request) from SCI (on ADC input)
 
Output O1: active = PERMISSION TO MOVE (hard signal)
Output O2: active = NOT DISCHARGE DETECTOR
Output O3: active = SHORT FILTER
Output O4: active = logical signal "permission to move" to SCI
 
There is also a second power supply which is used: it is the black lemo cable which goes from the grey distribution box of LV 6V (next to the 12V supply) onto the distribution box. This power supply is used for 2 purposes: to power the HV relays in the HV filter (through the cables "decharge" and "charge rapide"), and to make a 5V TTL signal which is send to the SCI.

MODIFICATION JAN/2005: There are now two lemo cables in parallel leading to relay 1 and relay 2; they are connected together with a lemo-T on both sides (on the controller in thebay, and inside the cone on the detector side). So where we read "the discharge cable" we mean the two cables together of course.  This has been done to secure better the link between the controller and the relays, and to reduce the voltage drop through the cables.

The controller has two independent functions: HV control and movement security.

• HV control: the controller receives from the SCI a TTL signal (logic high) when there is a request to move. It then UNPOWERS the "decharge" cable (by opening the relay connecting it to the 6V supply). This should then have as an effect that relay 2 of the HV filter grounds the detector and discharges it. On the HV display nothing will change, as the HV filter separates the supply from the detector. This situation is maintained as long as the SCI keeps this request signal high. (The voltage is measured with an ADC, as there are no TTL inputs on the controller.) When this signal goes low again, "decharge" is connected again to the 6V supply, and the detector is not grounded anymore (can charge again). A short delay later, during 6 seconds, relay 1 is closed, so that the detector can charge up rapidly. After these 6 seconds, relay 1 is opened again, so that the filter is active again.

  There are some extra safety delays present in the scheme:

  • in the safety system: when low voltage on the detector is detected, there is still a 2 second delay before permission is given to move (O1), and 2.4 seconds before the logical signal for permission (O4) is sent to the SCI. These 2 seconds serve as extra discharge time (safety) for the detector. The 0.4 extra seconds allow the SCI to assume hard permission granted when they receive logical permission. However, when a high voltage is detected, immediately the logical signal for permission is withdrawn (O4) and 100 ms later the hard permission is denied (O1). The 0.1 seconds serve to signal first logically to the SCI that permission will be denied before denying it actually.
  • in the HV control system:
    • when a movement request is detected, an eventual fast charging (O3) which could still be ongoing (within 6 seconds from the previous cycle) is immediately stopped, and 0.5 seconds later the detector is discharged (O2 desactivated). These 0.5 seconds serve as buffer between opening relay 1 and closing relay 2, so as not to have sparks on them. Note that in most situations, the request for movement will come much later than 6 seconds before the preceding cycle and so relay 1 will already be open (and the filter not shorted).
    • when an end of movement request is detected, a small delay of 0.2 seconds is introduced before the detector is "ungrounded" (opening of relay 2, O2). This very small delay serves the purpose that at the end of the movement there might still be some mechanical vibrations, so before recharging (even slowly) the detector we wait a very small amount of time. After 1 second, we activate the fast charging (relay 1 - O3) during 6 seconds (unless interrupted by a new request for movement). The delay of 0.8 seconds between the opening of relay 2 (ungrounding) and the closing of relay 1 (fast charging), again to avoid sparks.
   
  Note that if the 6V supply is not present, because, for example, the power supply is off, or the lemo cable is not connected, then the detector will be discharged, but relay 1 will be open (so the filter will be present between the HV supply and the short circuit on the detector). This means that the HV supply will not be shortened, and you will read 1515V on the supply, even if the detector is discharged.
 
There is however a way to find out if the detector is discharged or not, and that is BY LOOKING AT THE CURRENT OF THE HV SUPPLY. (Turn the round knob to "I" instead of "V"). 



• The movement security. When the "measure" voltage is over a certain threshold, then the power is cut from the relay that will activate the motors (that's the big grey cable). So it is impossible to move the detector when there is HV on it. This is the essential part of the security system.
  There is a pull-up resistor to the power supply of the controller onto the "measure" connector, so if you remove the "measure" cable (or it is cut), then automatically the unit assumes there is HV on the detector and the permission to move is denied.

  THIS CAN BE A CAUSE OF TROUBLE: if there is a bad contact on this cable, the SCI will not get permission to move, even if the detector is discharged.

  Apart from this "hardware" permission, there is also a TTL signal which is send to the SCI, 100 ms after we grant the hardware permission: the signal goes high 100ms after we close the permission contact, and stays high until the "measure" voltage goes beyond its threshold again.

  Note that this "TTL" level is derived from the 6 V supply, using a diode, so we get 6V - 0.7V = 5.3V out (which is an acceptable TTL level).

To check whether the detector is at 0 V, go to the HV supply (which should always indicate the nominal voltage, 1515V), and turn the knob of the readout to "I". If the reading is something like 0.02, then the detector is normally under tension (the small current of 15 microamps is the result of the voltage divider of 100M). If the reading is something like 0.84, then the detector is discharged and the current is the one flowing through the 1M8 RC filter.

Check whether the supply is in "current limit mode". If not, there is a problem with the HV supply (or the voltage is set to 0 V; set it to 1515 V).

Check that the lemo connectors of the double cable with "decharge" (and also the blue scotch on it) is well connected to the unit. Unplug it and plug it in again. If that doesn't help, plug that double cable in the free 6 V outlet in the grey box next to the 12V supply (so we force the discharge relay to open). This should work. IF IT DOESN'T, go to topic "Discharge relay is stuck". If this allows the HV on the detector to go up again (current falls from 0.84 to 0.02), check whether the controller is working (see topic "The controller is not working").

If the controller is working, the relay is not stuck but it doesn't work when you put it onto the control unit, check whether you are allowed to move the detector when it is discharged (0.84 current). If that's the case, you can go to topic "Partially manual mode". If this is not the case, go to topic "Live dangerously".

Check whether the red and green leds next to the motor controllers (on top of the led box) in the control room are lit when you ask to move. If they are lit, it is a problem with the SCI, not with the detector.

Unplug and replug the cable "mesure" on the control unit. Measure de voltage on the that cable (by finding a lemo base and a multimeter). You should read the voltage on the detector, divided by 1000. So if you find about 1.5 V, then the detector is NOT at 0 V; if you find a very low voltage, then this should normally work; in this case, try to measure the resistance of the wire: you should find 100 kOhm (if not, there's something wrong with with this cable). Warning: only measure that resistance when all HV has been removed! Go to topic "Live dangerously".

Note: it is normal that you have to wait for 7 seconds before you are allowed to move the detector.

Unplug and plug in the LEMO cables from the SCI on the controller (lemo 1 and lemo 2). Check whether the controller is working (see "the controller is not working"). You might ask an SCI responsible (e.g. Jean Vidal) to check whether his signals arrive at the controller or not.

Check whether the 6 V supply lemo cable is well connected to the grey box next to the 12 V supply. Check whether the +6 V supply of the amplifiers is up and running (has 6.06 V or so on its display).

If you do not succeed, go to topic "partially manual mode".

Take the box and look on the LCD display. In the upper left corner, there should be a small "wheel" turning. If not, try to switch it on again. If the display is blank, check whether the two 12 V adapters on the back of the rack (go around, open the back door) are plugged in. It might help to unplug them, wait a minute, and plug them in again to restart the controller.

There is a spare controller in the cabin somewhere on the shelves.  With some work, you can replace the existing controller by the spare one.  Normally this should be done by a knowledgeable person (P. Van Esch for instance).

If you do not succeed, go to topic LIVE DANGEROUSLY.

If the discharge relay is stuck even after plugging it into the 6 V supply, try to find a power supply with banana connectors, and take the small tool (a piece of pcb with some lemo bases on it with a black and a red banana cable). Take the "discharge" cable (with the blue label) and plug it into the tool, connected to a supply. Put 7 or 8 V on it: hopefully now the relay gets released.

Test whether it works now on 6 V using the 6 V supply. If it is ok, we can go back to normal automatic mode. If not, go to topic "partially manual mode".

There are different levels, according to the problem at hand. The simplest case is that we leave the "discharge" cable plugged into the control unit, but we switch off the HV supply and wait a minute or so before giving movement commands.

We can also plug the "discharge" cable into the 6 V supply. Finally, we might need to plug it into the small tool (from topic 13.2.5) and keep it at 7 or 8 V.

Check whether you can move the detector after waiting for a minute after switching off the HV. If not, go to topic 13.2.2. In this mode, the controller still functions as a security device, so security is still guaranteed.

This mode of operation is highly discouraged, because one single error can easily kill a 2,000,000.00 Francs detector.

You are highly discouraged to do this. Is it really worth to risk to sacrifice the whole detector just for a few hours/days of beam?

OK, you still want to do so ?

Find a Lemo T-piece and connect lemo 1 and lemo 2 from the SCI together.
Take the "decharge" cable and the "charge rapide" and plug them both into the 6V supply.
Switch off the HV.
Unscrew the big connector and put the terminator plug on it (that's the big loose connector somewhere in the detector-electronics cabinet).
Make ABSOLUTELY SURE that you switch off the HV and wait for a minute before launching a script or a command that will move the detector, or before touching the manual mode commands.
Unscrew the plug from the big cable: the detector is again not allowed to move.
You can now switch on back the HV, the detector will slowly charge up.

IT IS YOUR PERSONAL RESPONSABILITY TO MAKE SURE THAT THE BIG PLUG IS NEVER ON THE CABLE WHEN THE HV UNIT IS POWERED.