Description of issue
Tushar Biswas was unloading his samples after completing his work and received an error message on the RAITH 150-TWO software (image unavailable) saying that there was a load-lock timeout. At this point the robot arm was half way through the VAT valve (isolates load-lock from main chamber) and would not move.
He then contacted me and submitted a request to report the problem.
Tentative solutions
I tried the solution left by Bob Breber (Transfer arm recovery (images and corrections to come)), but it did not work. The computer seems to be unable to communicate with the load-lock controller.
After contacting RAITH support, Bob Urban called and we tried debugging any issues with the controller. We took it out of the rack and opened it up to test any damaged components. Fuses are good and lights turn on when the system is powered up, and we could not find anything obviously out of the ordinary.
Bob then suggested trying a manual override switch to control the arm motor independently. He thought that we had it with us, but I could not find it so he's sending one to us, which we will have to return after we are done. This will allow us to both test the motor itself and, if it works, to recover the samples currently stuck inside the system. The switch is expected to arrive sometime on Friday, .
Update
We've finally received the manual override switch, after a big delay by FedEx. I'll use it under Bob's "supervision" over the phone tomorrow, April 13 at 8:30 am MDT.
I've already identified what connector will be used under the loadlock (silver case, grey cable, unlabeled, connected to "Loadlock Motor" on the back of the Loadlock Controller.
Update
Called Bob Urban and we managed to get the system working again.
We disconnected the motor from the Loadlock Controller and connected Bob's manual override switch. With the switch we were able to move the arm all the way to the loadlock, allowing us to close the VAT valve (loadlock-chamber).
After that we had some issues trying to vent the loadlock through the software. It seemed that the controller was preventing any attempt to work on the loadlock because it was unaware of the system's state. We decided to loosen a couple of the screws holding the vent valve and manually vent the loadlock by letting air in. At this point I started hearing the rough pump making noise, which meant that the pump valve was still open. After closing the vent valve we tried venting the loadlock through the software again, but it would only open the vent valve momentarily and immediately close it again. So we tried venting with air again by loosening the screws, which worked nicely this time.
With the loadlock open, I recovered Tushar's samples. I'll send him a message so he can come and pick them up.
Bob asked me to check a couple of things. First, the belts that move the arm in and out of the loadlock, which are in good condition and in proper place (the fact that the override worked is another good indicative). Then Bob asked me to very slowly and carefully pull the arm back towards the end of the loadlock (away from SEM), until we saw the software recognize contact with the end-switches — this had to be done very carefully and with the arm almost at the end of course, otherwise we could break the belts. From there we were able to reinitialize the system and everything started working again.
Bob said that we can keep the override switch. The package came with an invoice and it only costs US$ 20.00, so we can easily pay for it if they require us to.
Update
Demetrios Kalamidas (Demetri, Raith Service Eng.) came to check the system and try to have it fixed.
After testing the system in the morning he concluded that the problem is not with the end-switches. All of them were properly activated and read when depressed manually. They were also properly read when activated by the robot arm.
We tried unsuccessfully to initialize the outer position a few times (inner was marked as done). The system would correctly move the robot arm out until the limit-switch was depressed, then move forward a bit to release it and keep only the outer position switch depressed. However, the software would not mark the initialization as complete.
We then tried to restart the controller by unplugging/plugging the power cable. After this we were able to initialize the outer position. Once the initialization completed, we successfully run a complete loading procedure — there was no timeout this time!
Demetri confirmed that the arm motion is looking good, and since it was abele to complete the loading procedure that the problem was not with the parts in the loadlock chamber, but rather in the controller itself.
He did note that we should replace the gas springs on the loadlock lid — it accidentally closed on his finger, but thankfully without hurting him seriously.
Since he may not be able to fix the controller, as it may need to be replaced, he offered us to run a PM on the column, aligning it and making sure that everything else is working well.
Update
Unfortunately the loadlock arm got stuck again, this time while Riley Endean (AQM) was trying to load his samples.
Gustavo de Oliveira Luiz tried recovering it following the steps performed with Demetri on , but had no success. This time the problem seems to have gotten worse, with the arm getting stuck both going into and out of the chamber (previously it would go in normally and get stuck only when moving out).
A quote for a new SPS controller was requested. Since the joystick is also malfunctioning, we also requested a quote for a new joystick.
Update
We received a new controller on .
The controllers were swapped with remote assistance from Demetri (over the phone) on . After installing the new controller, the system continued to present the same problems.
Bob Urban called and guided Gustavo de Oliveira Luiz for a few tests on . We used an oscilloscope to monitor the test pins A, B and MOT. From the tests we made it was clear that the motor sends a few signals back to the controller as it is moved, which was made obvious by the A/B signal changing as we manually pushed the robot-arm. The MOT signal didn't seem to change when manually moving the arm, but it does got up when the controller moves the motor — it seems to be a simple signal to indicate if the motor is being driven or not.
Interestingly enough, the MOT signal when the motor is NOT moving is ~0 V on the old SPS controller and but ~1.5 V on the new one. Bob didn't expect any signal, but since both SPS controllers were working the exact same way we concluded that this was not the issue. Our conclusion was that something was wrong with the motor.
We first tried checking for any wires touching the motor case inside the loadlock chamber. Nothing out of place was found. Then, we tried increasing the current supplied to the motor by adjusting a potentiometer inside the SPS controller, again to no avail.
After these tests, Bob concluded that the motor must be the problem and we requested a quote for a new one. The SPS controller replacement will be returned — we are waiting for the new motor to arrive so we can make sure the problem is fixed, then we still have to test the new joystick such that we can send the old joystick back as well, all in one package.
A new motor was ordered on and haven't received it yet as of today.
Update
We received a motor on , however it was the wrong model. Our system has a step motor installed and RAITH sent us a DC motor instead. The problem was verified and RAITH dispatched the correct motor on .
The correct motor was delivered on and we scheduled the remote assisted installation for .
On Adam Pettigrew and I worked to replace the faulty motor, while Bob Urban (RAITH) was on the phone to give us guidance. After turning the SPS controller off, we successfully accessed the motor and removed it following the guide Bob sent to us. This required opening the bottom cover of the loadlock, detaching the motor block from the belt (there is a coupler that allows us to remove the motor without disturbing the belt), detaching the encoder block and unsoldering the 4 wires from the motor.
After the motor block was free, we took note of relative positions of some parts with a caliper and continued disassembling it. This was no simple task at first, since Bob wanted us to work the coupler cap and encoder wheel out of the shaft, but they were both stuck due to bur on the shaft. We successfully removed the coupler cap, but the encoder wheel could not be removed. The main issue was a lack of gripping places and the fact that the encoder wheel is fragile and we didn't have a replacement. Some minor damage happened to the block due to the use of a vicegrip, but this does not affect operation (it's just an aluminum block where the motor is mounted). The faulty motor eventually broke as we used it as a gripping point. Since the motor had no value to RAITH anymore, we decided to take it to the subfab, cut the shaft and punch the stuck piece from the wheel. We could probably have saved the motor had we tried this from the beginning.
Once the encoder wheel was free, we were able to remove the motor from the aluminum block and install the new one. Installation went smoothly, and we ended up replacing the heat-shrinks because it was the easiest thing to do. We made sure all parts were back at the correct position, checking that the encoder wheel would move freely inside the encoder block and that the coupler would properly engage the belt.
After installation, the system was tested a few times and no issues were observed. The system was marked as UP on LMACS and a message sent to all users.
The only remaining issue with the loadlock is that the gas springs are weak and need to be replaced. New ones were ordered on but the lead time was 14 weeks (ETA: ). Users have been made aware of the issue and asked to be careful with the lid.
Vacuum monitor
Since the system has the VAT valve open and the column valve close, I'll use this space to monitor it so we can make sure the system is safe.
| Date | System vacuum (mbar) | Gun vacuum (mbar) | Column valve status |
|---|---|---|---|
| 1.89e-6 | 4.61e-9 | CLOSED |
| 1.51e-6 | 4.56e-9 | CLOSED |
| 5.47e-7 | 3.10e-9 | OPEN |