-
Notifications
You must be signed in to change notification settings - Fork 4
Progress Log for ARTN: Kuiper Roboticization
The purpose of this page is to keep a log of progress, preferably on a nightly basis.
Jan 9, 2019 --
DS: Caught up on ARTN with Scott and Sam.
Wrote email to Lesser about being able to start Azcam in ARTN mode programmatically.
Started slack page.
Started log page (this page!).
Scott and Sam: Began work on message page to let us know the status of RTS2. This will appear in the RTS2 queue manager.
Jan 10, 2019 --
First thing to check: Does the flat script run even if the 'After Flats' button is pressed? -- This question was not answered due to the following problem.
Early in the evening we are running into some issues with the ARTN web page -- we are thinking that it has to do with hitting the database too much. After some discussion, we have decided that bigpop is not up to the job. We are going to spec out an appropriate ARTN computer for the mountain.
update: I found a bug with the web page that I introduced today. The buggy code was trying to query the messages 100 times a second. I have limited that to once every 5 seconds.
Scott streamlined the start/ending of the night. The buttons and procedure have been updated in the wiki.
Sand edited the wiki a little -- it turns out we were missing a step where we had to load the rts2 header into AZcam. (We were using the previous observer's fits template.). Sand also wrote an 'End of Night' procedure.
Scott found some tell-tale signs of when the camera driver will fallout. He elaborated on the Drivers Falling Out Issues
Jan 24, 2019
-
Scott fixed a bug that had plagued us for a while. Flat images now display in the quicklook ds9. Nice work, Scott.
-
We observed some SN2018ivc. That went smoothly, and so we decided to break something.
-
We compile Petr's new code that was meant to fix the recurring camera driver dropout issue. So far, things are not working cleanly -- all output images have pixel values of 512. Will call Petr shortly. UPDATE: we were not able to get a hold of him. Will try to have lunch with him tomorrow.
-
We set up the rsync cron job to send all ARTN data down to scopenet under /data1/artn/rts2images/. That was easy, but also a big victory.
Feb 5, 2019
Scott & Sand went up the mountain during the day.
Sand worked on the ARTN Equipment Needs and Budget document. Focus was on humidity and wind sensor.
Scott has reworked things so that we no longer have to fiddle with the 'After Flats' button. That will need to be tested on sky during the next run.
Feb 19, 2019
Scott is working on infrastructure to link up with Phil's ARTN Observation Request Portal.
Dave messed around with astroplan and did some primitive scheduling of a telescope.
Night of Feb 19, 2019
Below is an email from Peter Milne:
- weather was a major issue....clouds early and blowing snow late. We were open for maybe 2 hours.
- We DID get focus images for Tim. In particular, we obtained a series of image pairs. Two each of -400, -500, -600, +400, +500, +600.....relative to our best estimates of the focus. The seeing was 2 arcsec and worse....so bear that in mind.
- The queue manager failed and we were not able to upload targets. Scott worked on this for a good long time, and then decided that since the changes were in support of Phil's software and we might never use the existing queue manager interface ever again....perhaps there was little point in continuing to try to fix that.
- Tim & Rachael were certified, after doing a complete lightning shutdown and cold startup after we closed.
All in all, some stuff was accomplished during a questionable night. Please be vigilant and aware that if it gets windy (15+ mph), blowing snow might be an issue....and we have to close when it is "snowing in the dome" due to wind-driven snow. Perhaps the snow on the surrounding trees has melted today.
Night of Feb 20, 2019 (from Phil Daly):
Night of Feb 20, 2019
e666ff0c9284f4a6477badee7cdee4e33a22ce1d
Comments by PND: bigpop does not support the new ARTN ORP as seemlessly as we would like. My code uses Python 3.7 (because of f-strings) and bigpop only has 3.5. Installed conda and setup a venv with 3.7 but - for reasons that remain obscure - bigpop thinks the port is in use when it's not (I tried 5000 and 7500). One thing we might all think about it where to host the portal as it's astronomer-facing and designed to support multiple telescopes. Nominally we can run one downtown and one at each telescope as, ultimately, they talk to a database. Comments welcome.
e666ff0c9284f4a6477badee7cdee4e33a22ce1d So, I ran the code on my laptop which was fine. There was some shenanigans with credentials to access RTS2 which was resolved. The /orp/observe page was updated to include links to RTS2 via Scott's example script. The page return indicates a problem on the RTS2 side (it returns a NoneType object which shouldn't happen). Scott will fix this and we will try the /orp/observe route sometime next week - day-time testing is fine. All other ARTN ORP routes worked fine. All in all, good solid progress towards implementing new functionality for ARTN.
Day of March 7, 2019
Scott & Phil are doing further integration work of AORP and ARTN-61 on the mountain.
March 30th, 2019
Continued work by Scott trying to figure out why the camera driver fails. Tonight the camera driver never really failed, but the executor failed often. I ran both the camera driver and executor in their own terminal. The executor would crash with a memory allocation when it was switching between targets. I installed valgind (a c/c++ debugger) to track down the malloc error, but the executor worked flawlessly after that. Very difficult bug to track down.
As I see it we have three main issues with RTS2 and nightly operation.
- AzCam gives an initialization error whenever RTS2 is restarted
- Camera driver crashes.
- Executor crashes.
I think 2 and 3 are related or maybe they are the same thing. I have another engineering night next weekend (April 7th.) I will continue this work then
April 7th 2019
I worked with Petr on the executor and camera driver crashes. Executor never crashed but eventually the camera driver did. Petr found a buffer overload in the camera driver and fixed it.
Since then I have been running a queue and no crashes (as of 11pm).
April 17th 2019
Scott S is working on opening/shutting the dome with RTS2.
P. Daly is working on the DNA -- Data notification agent.
Dave S has uploaded some candidates to AORP via the file upload form.
Dave S has updated the 'Putting objects in the queue' section of the wiki.
May 16th 2019
P. Daly is working on incorporating nonsidereal tracking into the AORP database.
S. Swindell is working on automating focus during start of night operations.
May 18th 2019
Fixed problem with dark. Merged Kuiper changes to upstream RTS2 and run the upstream RTS2.
e666ff0c9284f4a6477badee7cdee4e33a22ce1d