Tool

Nanoscribe Photonic Professional GT

Issue

On 12 May 2023 night, a campus wide power outage affected the clean room, powering down equipment. When the Nanoscribe PPGT+ was powered up, the Powerscaling calibration was capping at 0.96, whereas the normal behavior is for it to cap at 1.35. The smaller than 1.0 limit would limit the available power to print to less than the nominal 50 mW max.

This issue is similar to what happened circa 20 Aug 2019.

Solution

Nanoscribe support was contacted and given a NanoWrite generated service report and a FFproRead log file. The latter is a laser specific software used to extract data from the laser and its components, such as the pump diode lasers health.

Alce Kamas sent the laser information to the manufacturer (Toptica) for diagnosis. They replied saying that the pump diodes should be OK, and that they believed a prism to be stuck at a local maximum instead of a global one. This is not a major issue, and Toptica said that it is know to happen sometimes after a power cycle of their laser.

The suggested solution was to force the prism to scan its entire range and recalibrate. This was done by Alec Kamas through TeamViewer on 18 May 2023 morning, and the problem was fixed with the Powerscaling calibration reaching 1.37.

Alec provided the following command to be run on NanoWrite (to avoid using the Toptica software he used) next time this happens:

SendCommand "laser" "ffpro.arm0.optimizer.scan()"

Tool

Nanoscribe Photonic Professional GT

Issue

On 12 May 2023 night, a campus wide power outage affected the clean room, powering down equipment. When the Nanoscribe PPGT+ was powered up, the galvo mirrors were malfunctioning, moving the laser only left-right not up-down.

Solution

Nanoscribe support was contacted and Alec Kamas provided the following steps to diagnose what part was faulty:

--- START---

Please wear gloves for all steps, as the metallic parts have sensitive coatings and some cables are reinforced with glass fibers. Always turn off the SCU before unplugging cables and wait for a minute before switching on again, i.e. do not rapidly switch the SCU on and off.

1. The SCU sits in the metal box shown in the picture (circled in red).
2. 
   For PPGT+/2 Systems the SCU sits in the safety electronics box:
   
   
   
3. Using a short Phillips screwdriver, remove the four screws on the sides of the box (a short screwdriver is recommended as there is little space between the microscope and the box; see picture below).
   
   For PPGT+/2 Systems, the safety electronics box is opened through five Torx T20 screws in these locations:
   
   NOTE the cover of the safety electronics box has an electronics unit and cables attached to it. Please leave all cables connected. You can stand the cover vertically so it sits within the bottom section of the safety electronics box. Take care not to accidentally press the emergency stop button!
   
4. With the SCU ON, check the status of LEDs on the SCU and send us a picture. After pre-start calibration and during normal operation, the "READY" LEDs should be lit green.
5. Turn the SCU OFF using the black power switch on the top left of the SCU box and wait for one minute.
6. Press the silver buttons below the CAL text for GSA and GSB. Turn the SCU back ON. The CAL LEDs should illuminate in orange while the galvo runs through a calibration routine, which takes approximately 90 seconds. The orange CAL LEDs will not turn off.
7. After this time, un-press/release the CAL buttons again. This should reset the galvo and it will perform the normal pre-start calibration. Wait until the "ready" LEDs turn ON. If no ready LEDs turn on, power down the entire system, restart and repeat steps 5 and 6. Observe the behavior after this restart and proceed with step 7.
8. Load a substrate and objective as per usual. You can use any substrate/objective. Find the interface and use the attached GWL file to continuously print circles. Note the behavior, which will be one of
   1. circles are printed correctly
   2. only a vertical or horizontal line is printed
   3. nothing is printed
9. Turn OFF SCU power, wait for one minute and switch the BNC cables on the front (marked as X-axis and Y-axis). In the picture below, swap the cable circled in red for the one circled in blue and vice versa.
10. Try to print the same test circles and note if the faulty axis is swapped. (If so, the input signal to the SCU is problematic and the SCU itself is probably fine.)
    
    
    
11. First revert step 8 (Turn OFF SCU beforehand) by returning the BNC cables to their original positions. In a similar sense to step 8, swap the silver cables with color-coded sleeves on the other side of the SCU as shown in the picture below. Note that both "red cables" should be swapped with both "black cables" and vice versa. The connectors have a different number of pins for the top and bottom connectors. DO NOT force connectors into the wrong socket!
    
    
    
12. Repeat steps 5 and 6 to fine calibrate with swapped axes. Turn the SCU back ON and note any change for the "ready" LEDs. Once again, try to print the test circles and observe if the faulty axis is swapped. (If the faulty axis swaps and no change occurred during step 8, the SCU channel board itself is the cause. If the faulty axis does not swap, the issue could be related to the galvo mirror hardware.)
13. Revert step 10 to return the silver cables to the correct positions.

---END---

Alec also suggested that leaving the system on overnight could fix the issue. However, this would indicate that one of the boards inside the SCU was faulty and the whole controller should be replaced.

After removing the lid as instructed, we found the LED next to "ready" was on for the GSB side but not for the GSA side (see below, GSC is not used).

Gustavo de Oliveira Luiz was able to perform steps 1-6, but without rebooting the system. It did not fix the issue and it was left powered up overnight in order to continue on the next morning. When arriving in the morning, both LEDs were found on (see below).

A quick test was done with the "infinite" circle provided by Alec and everything worked as expected.

Since the system returned to normal operation the tests could not be completed (they require the system not to work in order to diagnose the issue). However, Alec suggested that the fact that the system came back by itself is a strong indicator that the SCU is faulty and should be replaced. A quote was provided and we are going to purchase the replacement. 

Tool

Laurell EDC-650 Spin Processor #2 (Primary Litho)

Date

13 Feb 2023 

Description

While transferring programs to the controller (Spin processor) using "Spin 3000", the software froze for a few seconds and resumed the process. After it was done, all programs have vanished from the controller and, when trying to upload them again the system showed an error message:

This indicated that the programs file on the computer (notebook) had some parameters that did not match the controller anymore, despite it being used many times before (including on the same day). The only option we had was to create a new file in which these parameters were made to match the current controller. However, at least 2 of these (Maximum Motor Temperature and Air Required) are important for us to have properly set, because they are interlock parameters (requirements) that would avoid operation in case they are not satisfied.

Some of the mismatching parameters, including these important interlock ones, could not be manually set either directly on the controller nor with "Spin 3000". So we contacted Laurell support to assist us, since these parameters are set at the factory so they must have another method to access them on the controller.

Paul Grasso was very unhelpful from the start, asking us to repeat the same thing over and over despite us telling him that it did not work. He insisted that the only way to configure the controller was with "Spin 3000", which was obviously not the case since it did not allow us to restore the parameters we wanted.

Finally, after stating that this was a rather unique situation that he's never seen before, he came up with another app, "SpinUpgrade.exe", that was supposed to do the job. The procedure is outlined in this document (ignore the steps for installing some other parts onto the spinner). The "org.spc" file contains the current (wrong) spin configuration, while the "new.spc" file contains the corrected parameters. To produce the former file, Paul asked me to crate a new ".spn" file with the current configuration, and he produced the latter file apparently simply changing the parameters as needed. These ".spc" files are identical to the ".spn" files in terms of content, so it seems that we should be able to easily produce them as needed (Paul Grasso never suggested it).

Running this "SpinUpgrade.exe" app fixed all of the mismatches but two: "Maximum Motor Temperature" and "Air Required". The former was actually misconfigured in the "new.spc" file, but the latter was properly set and still did not alter the controller. This led to another series of exchanges with Paul, who insisted that "Spin 3000" was supposed to adjust these parameters. In his words: "Spin 3000 configures the controller.  It is not the other way around." — I would not even have contacted them if that was the case. He even suggested creating a new notebook after the mismatch error, then opening the new ".spn" file with a text editor to change the "Air Required" parameter and uploading it to the controller. This, of course, did not work and only caused the mismatch error once again — this is the problem after all.

After the absurd suggestions, implicitly implying some incompetence on our side, Paul said that their software engineer came up with a new "SpinUpgrade.exe" app which should now work. In fact, this time the patched app worked and the controller was once again properly configured.

Other comments

• The same issue also happened a few days before with the spinner on the Spin Bake Station #1 (Left). In this case only the "Nozzle" configuration was lost on the controller, and since there are no nozzles and this is not an interlock parameter, a new file matching the new controller configuration was created to we can continue using the system.
• There is a chance that the "SpinUpgrade.exe" app is universal and should work on all our Laurell spinner systems (at least the newer ones, not sure about the older one in the fume hood).
  • I asked this to Paul in the context of the spinner that suffered the same issue, but all he said is that we should not worry about it because the parameter that changed is not relevant. He never confirmed or denied if the same app would work or if we needed one for each tool.
• The experience with Paul was very unsettling, and we will think twice before requesting assistance again. Aaron said that he will follow up with Laurell's sales representative to report this, perhaps we can have someone else assigned as our main support contact.
• Besides the controller configuration issues, we also asked Paul about the BT not working on the spinners. His response was that it is not supposed to because we purchased the cables for physical connection, which caught us very much by surprise.
  • Aaron will also discuss this with Laurell in a separate thread (mixing this up with the other issues would clearly be a mistake).

People involved

Gustavo de Oliveira Luiz (author)

Aaron Hryciw 

RAITH support

Description of issue

Tushar Biswas was unloading his samples after completing his work and received an error message on the RAITH 150-TWO software (image unavailable) saying that there was a load-lock timeout. At this point the robot arm was half way through the VAT valve (isolates load-lock from main chamber) and would not move.

He then contacted me and submitted a request to report the problem.

Tentative solutions

I tried the solution left by Bob Breber (Transfer arm recovery (images and corrections to come)), but it did not work. The computer seems to be unable to communicate with the load-lock controller.

After contacting RAITH support, Bob Urban called and we tried debugging any issues with the controller. We took it out of the rack and opened it up to test any damaged components. Fuses are good and lights turn on when the system is powered up, and we could not find anything obviously out of the ordinary.

Bob then suggested trying a manual override switch to control the arm motor independently. He thought that we had it with us, but I could not find it so he's sending one to us, which we will have to return after we are done. This will allow us to both test the motor itself and, if it works, to recover the samples currently stuck inside the system. The switch is expected to arrive sometime on Friday, 08 Apr 2022.

Update 12 Apr 2022 

We've finally received the manual override switch, after a big delay by FedEx. I'll use it under Bob's "supervision" over the phone tomorrow, April 13 at 8:30 am MDT.

I've already identified what connector will be used under the loadlock (silver case, grey cable, unlabeled, connected to "Loadlock Motor" on the back of the Loadlock Controller.

Update 13 Apr 2022 

Called Bob Urban and we managed to get the system working again.

We disconnected the motor from the Loadlock Controller and connected Bob's manual override switch. With the switch we were able to move the arm all the way to the loadlock, allowing us to close the VAT valve (loadlock-chamber).

After that we had some issues trying to vent the loadlock through the software. It seemed that the controller was preventing any attempt to work on the loadlock because it was unaware of the system's state. We decided to loosen a couple of the screws holding the vent valve and manually vent the loadlock by letting air in. At this point I started hearing the rough pump making noise, which meant that the pump valve was still open. After closing the vent valve we tried venting the loadlock through the software again, but it would only open the vent valve momentarily and immediately close it again. So we tried venting with air again by loosening the screws, which worked nicely this time.

With the loadlock open, I recovered Tushar's samples. I'll send him a message so he can come and pick them up.

Bob asked me to check a couple of things. First, the belts that move the arm in and out of the loadlock, which are in good condition and in proper place (the fact that the override worked is another good indicative). Then Bob asked me to very slowly and carefully pull the arm back towards the end of the loadlock (away from SEM), until we saw the software recognize contact with the end-switches — this had to be done very carefully and with the arm almost at the end of course, otherwise we could break the belts. From there we were able to reinitialize the system and everything started working again.

Bob said that we can keep the override switch. The package came with an invoice and it only costs US$ 20.00, so we can easily pay for it if they require us to.

Update 23 Jun 2022 

Demetrios Kalamidas (Demetri, Raith Service Eng.) came to check the system and try to have it fixed.

After testing the system in the morning he concluded that the problem is not with the end-switches. All of them were properly activated and read when depressed manually. They were also properly read when activated by the robot arm.

We tried unsuccessfully to initialize the outer position a few times (inner was marked as done). The system would correctly move the robot arm out until the limit-switch was depressed, then move forward a bit to release it and keep only the outer position switch depressed. However, the software would not mark the initialization as complete.

We then tried to restart the controller by unplugging/plugging the power cable. After this we were able to initialize the outer position. Once the initialization completed, we successfully run a complete loading procedure — there was no timeout this time!

Demetri confirmed that the arm motion is looking good, and since it was abele to complete the loading procedure that the problem was not with the parts in the loadlock chamber, but rather in the controller itself.

He did note that we should replace the gas springs on the loadlock lid — it accidentally closed on his finger, but thankfully without hurting him seriously.

Since he may not be able to fix the controller, as it may need to be replaced, he offered us to run a PM on the column, aligning it and making sure that everything else is working well.

Update 07 Jul 2022 

Unfortunately the loadlock arm got stuck again, this time while Riley Endean (AQM) was trying to load his samples.

Gustavo de Oliveira Luiz tried recovering it following the steps performed with Demetri on 23 Jun 2022, but had no success. This time the problem seems to have gotten worse, with the arm getting stuck both going into and out of the chamber (previously it would go in normally and get stuck only when moving out).

A quote for a new SPS controller was requested. Since the joystick is also malfunctioning, we also requested a quote for a new joystick.

Update 11 Aug 2022 

We received a new controller on 21 Jul 2022 .

The controllers were swapped with remote assistance from Demetri (over the phone) on 22 Jul 2022 . After installing the new controller, the system continued to present the same problems.

Bob Urban called and guided Gustavo de Oliveira Luiz for a few tests on 25 Jul 2022. We used an oscilloscope to monitor the test pins A, B and MOT. From the tests we made it was clear that the motor sends a few signals back to the controller as it is moved, which was made obvious by the A/B signal changing as we manually pushed the robot-arm. The MOT signal didn't seem to change when manually moving the arm, but it does got up when the controller moves the motor — it seems to be a simple signal to indicate if the motor is being driven or not.

Interestingly enough, the MOT signal when the motor is NOT moving is ~0 V on the old SPS controller and but ~1.5 V on the new one. Bob didn't expect any signal, but since both SPS controllers were working the exact same way we concluded that this was not the issue. Our conclusion was that something was wrong with the motor.

We first tried checking for any wires touching the motor case inside the loadlock chamber. Nothing out of place was found. Then, we tried increasing the current supplied to the motor by adjusting a potentiometer inside the SPS controller, again to no avail.

After these tests, Bob concluded that the motor must be the problem and we requested a quote for a new one. The SPS controller replacement will be returned — we are waiting for the new motor to arrive so we can make sure the problem is fixed, then we still have to test the new joystick such that we can send the old joystick back as well, all in one package.

A new motor was ordered on 26 Jul 2022 and haven't received it yet as of today.

Update 29 Aug 2022 

We received a motor on 17 Aug 2022, however it was the wrong model. Our system has a step motor installed and RAITH sent us a DC motor instead. The problem was verified and RAITH dispatched the correct motor on 19 Aug 2022 .

The correct motor was delivered on 25 Aug 2022 and we scheduled the remote assisted installation for 29 Aug 2022.

On 29 Aug 2022 Adam Pettigrew and I worked to replace the faulty motor, while Bob Urban (RAITH) was on the phone to give us guidance. After turning the SPS controller off, we successfully accessed the motor and removed it following the guide Bob sent to us. This required opening the bottom cover of the loadlock, detaching the motor block from the belt (there is a coupler that allows us to remove the motor without disturbing the belt), detaching the encoder block and unsoldering the 4 wires from the motor.

After the motor block was free, we took note of relative positions of some parts with a caliper and continued disassembling it. This was no simple task at first, since Bob wanted us to work the coupler cap and encoder wheel out of the shaft, but they were both stuck due to bur on the shaft. We successfully removed the coupler cap, but the encoder wheel could not be removed. The main issue was a lack of gripping places and the fact that the encoder wheel is fragile and we didn't have a replacement. Some minor damage happened to the block due to the use of a vicegrip, but this does not affect operation (it's just an aluminum block where the motor is mounted). The faulty motor eventually broke as we used it as a gripping point. Since the motor had no value to RAITH anymore, we decided to take it to the subfab, cut the shaft and punch the stuck piece from the wheel. We could probably have saved the motor had we tried this from the beginning.

Once the encoder wheel was free, we were able to remove the motor from the aluminum block and install the new one. Installation went smoothly, and we ended up replacing the heat-shrinks because it was the easiest thing to do. We made sure all parts were back at the correct position, checking that the encoder wheel would move freely inside the encoder block and that the coupler would properly engage the belt.

After installation, the system was tested a few times and no issues were observed. The system was marked as UP on LMACS and a message sent to all users.

The only remaining issue with the loadlock is that the gas springs are weak and need to be replaced. New ones were ordered on 08 Aug 2022  but the lead time was 14 weeks (ETA: 14 Nov 2022). Users have been made aware of the issue and asked to be careful with the lid.

Vacuum monitor

Since the system has the VAT valve open and the column valve close, I'll use this space to monitor it so we can make sure the system is safe.