Add optional test to download all files from the server

[teutoburg]

The recent hiccup with outdated file hashes would have been discovered if we had that. Certainly shouldn't run every time to avoid rate limiting. Maybe can be triggered whenever a file in ./database is changed (which says nothing about the status of the file on the server, but whatever) or manually. Should be considered in the whole ScopeSim_Data cache location issue, to avoid just using those and not discovering hash mismatches.

[hugobuddel]

Maybe do something entirely different than these hashes?

I've created a "Stable" directory in https://github.com/AstarVienna/METIS_Environments/tree/main/Stable which installs stable versions of all our packages in a configuration that is known to work well together. So we always have a stable working system to fall back on.

But it now does not work anymore because it errors out with

ValueError: SHA256 hash of downloaded file (index.yml) does not match the known hash: expected 2010d9054490bad92ef4c216cde3c721c3f2dd785017476332d67dbdaa523ebb but got 140d505e48e14502c28b4d595eba6cb5ed20f4d0ceace2cfadfedb9826582cc2. Deleted download for safety. The downloaded file may have been corrupted or the known hash may be outdated.

That is 'fixed' in d577156, but that still breaks this 'stable' configuration.

Maybe we should version these files as well or something like that? Breaking existing code so arbitrarily should not be possible.

Or perhaps just put the entire database in the package? It is only a gigabyte or so right?

[teutoburg]

Well, the "working combination" is also the idea behind the poetry.lock files, I don't know what's wrong with just using that but I don't mind if you're using something else somewhere else.

The hashes broke because of a dumb combination of 1) us not using pooch with version control (it's complicated, basically our way of using "dev versions" is wrong, and I need time that I don't have to come up with something better), as they strongly recommend, and 2) myself being incredibly stupid by pushing to our server from the wrong branch.

I think the setup we have is very solid if we use it right. Also, I'm prototyping a slightly modified way of doing something similar in ScopeSim-Targets, which seems to solve a few of these issues. But I didn't have time to properly test it in the last two weeks, that's why I haven't told you and that's also why the cache dir PR in astar utils hasn't made progress since. But I'm getting there.

In summery, I do believe that I have a solid plan on how to prevent these things in the future, that works with our existing infrastructure...

[teutoburg]

I realize I should elaborate on two points here:
Firstly of course one issue with the lock files is that we currently don't have one for the "full stack". That could be added right now in ScopeSim_Data (and would solve some problems there...), but I'm not sure if that's the right place. Ultimately, ScopeSim-the-Application would contain such a top-level lock file.

Secondly on not doing versions right: What I mean is that we're using prerelease version numbers for something thats should be handled with dev version numbers. Embarrassingly, I only recently learned that those are two separate things. I really want to come up with a solid dev version scheme for the ScopeSim Ecosystem some time soon. But I feel like that's a battle I have to fight myself, because there's too much crap out there that really doesn't suit our overall situation and workflow. Perhaps I can use one of my upcoming long distance train rides to think about this...

[hugobuddel]

Well, the "working combination" is also the idea behind the poetry.lock files,

That would still have broken this use case. Because then I would have specified a specific version of ScopeSim that I would have verified to work, and that would have a specific version of speXtra in its lock file. That specific version of speXtra would have been broken when the index.yaml on the server was changed, breaking the "verified to work" version of ScopeSim.

One thing that I don't understand is that I actually include the cache (so with the known-to-be-working version of those index.yaml files) in that repository. Actually to make it work without a network, but it should have prevented this problem too. But for some reason it still downloads the files again and then complain that the downloaded file is wrong, while the correct (=old) file is already there.

I don't know what's wrong with just using that but I don't mind if you're using something else somewhere else.

That is besides this issue, but since you mentioned it, let me elaborate anyway.

Ultimately what is wrong with poetry.lock is that it is language specific, and thus cannot capture the full stack. For example, due to 'reasons' (me being stupid), I still need to have 1 conda package on the server side of the archive. Plus all the operating system dependencies. And the irdb, and other random crap. At that point, the poetry lock is not sufficient anymore anyway.

Practically, I also have a similar "head" setup, where I clone the tip of the main branch for all our repos. Now those two setups are symmetric, in one of them I have tags, in the other the main branch.

Ultimately, I want to replace all of this with Guix packages, at least for my own setup. Those are perfect. The package manager of the operating system is the same as you'd use as a user.

Also, in Guix, there is little difference between installing the package (guix package -m mymanifest.scm), creating a virtual environment (guix shell mymanifest.scm), and a container (guix shell -C mymanifest.scm). Even running a VM is similar, guix system vm .... So the transition from installing arbitrary packages -> virtual environments -> containers -> virtual machines -> physical machines (guix deploy) is very natural, so special software needed.

In summery, I do believe that I have a solid plan on how to prevent these things in the future, that works with our existing infrastructure...

Great! But..

A technique I used during my PhD is first answer "Where am I going to write about this in my thesis?" before performing any action. Do you know where this ought to go in your thesis?

How does this solid plan lead to (better) science? For sure it does, but that science-connection should be made explicit, and that can be hard. I'd put it in the reproducibility (as required for science) category.

[teutoburg]

It's always fun how these issues tend to "escalate" (not in a bad way) from their original topic ^^

That would still have broken this use case.

Yes.

But for some reason it still downloads the files again and then complain that the downloaded file is wrong, while the correct (=old) file is already there.

I'm not entirely sure this is the original intent (from the pooch devs), but I can see one potential issue with just blindly using the cached file: What if the file actually does change, say we renormalize the flux column in some of the spectra (which actually might be necessary at some point). Then how would we ensure that the new version is used and not the old cached one? If used with versioning, pooch will create a separate cache for each version (or was is for each major? not sure). That would actually solve the issue I just described.

Ultimately what is wrong with poetry.lock is that it is language specific, and thus cannot capture the full stack. For example, due to 'reasons' (me being stupid), I still need to have 1 conda package on the server side of the archive. Plus all the operating system dependencies. And the irdb, and other random crap. At that point, the poetry lock is not sufficient anymore anyway.

Ok yeah, I see your point, especially about the IRDB. In places like the METIS_Simulations repo, the latter was supposed to be solved by running download_packages() with a specific version of each package (and not the default "latest stable"). But now there are two places there where the download is called, and the one that's actually used does not pin versions. I complained about this multiple times recently, but that's a separate issue.

Ultimately, I want to replace all of this with Guix packages, at least for my own setup. Those are perfect. The package manager of the operating system is the same as you'd use as a user.

I don't know about those, although I know you have mentioned it in the past. I suppose that wouldn't work for a windows use like me 🙂

What I am a bit annoyed with recently tbh is the wild explosion of different systems we recently started using elsewhere. IIRC, we now use uv, docker, podman and what not, besides poetry. Don't know if it's just me getting old, but it feels like we're just jumping onto every shiny new thing because it promises to solve some issues that we might have been perfectly able to solve with what we have, but now we depend on 5 new things. I suspect some of those decisions might have been influenced by AI, but that's just a feeling. Don't get me wrong though, I'm not at all saying our Guix falls under that!

A technique I used during my PhD is first answer "Where am I going to write about this in my thesis?" before performing any action. Do you know where this ought to go in your thesis?

Kieran did encourage me in the past to write something about what he likes to call "PhD-ware". As in, principles of software development and lessons learned, in the context of Astronomy. I like that idea, especially given how much crap I've seen over the year, just at our institute alone. Given time constraints, this will probably just become an appendix to the thesis for now, but I might later publish it somewhere.

That said, without pointing fingers now, I did very recently encounter the very person who talked about all of this doing some very questionable commits somewhere. Pretty sure it was AI, just from the phrasing of things there. Oh and we almost ended up with en and em dashes in a YAML file, I wonder how they're intentionally typing those on a German keyboard 😶

[hugobuddel]

It's always fun how these issues tend to "escalate" (not in a bad way) from their original topic ^^

But for some reason it still downloads the files again and then complain that the downloaded file is wrong, while the correct (=old) file is already there.

I'm not entirely sure this is the original intent (from the pooch devs), but I can see one potential issue with just blindly using the cached file: What if the file actually does change, say we renormalize the flux column in some of the spectra (which actually might be necessary at some point). Then how would we ensure that the new version is used and not the old cached one? If used with versioning, pooch will create a separate cache for each version (or was is for each major? not sure). That would actually solve the issue I just described.

There are two things, the idea behind 'Stable' in METIS_Environments should not ever change automatically; only deliberately. Because even an positive change could break something. So there is always something ready to demo. It never reached that stage, but that is fine. So it is also fine that it broke. I should have been more diligent.

The other, more important thing, is that existing installations from other people might now be broken. That is probably indeed best fixed by this versioning mechanism, as I understand it from your description. And it is fixed by upgrading, which people should do anyway. So there is no immediate problem.

Ultimately what is wrong with poetry.lock is that it is language specific, and thus cannot capture the full stack. For example, due to 'reasons' (me being stupid), I still need to have 1 conda package on the server side of the archive. Plus all the operating system dependencies. And the irdb, and other random crap. At that point, the poetry lock is not sufficient anymore anyway.

Ok yeah, I see your point, especially about the IRDB. In places like the METIS_Simulations repo, the latter was supposed to be solved by running download_packages() with a specific version of each package (and not the default "latest stable"). But now there are two places there where the download is called, and the one that's actually used does not pin versions. I complained about this multiple times recently, but that's a separate issue.

Yes, so for my archive setups, I bypassed that download from METIS_Simulations. But it is better now.

Ultimately, I want to replace all of this with Guix packages, at least for my own setup. Those are perfect. The package manager of the operating system is the same as you'd use as a user.

I don't know about those, although I know you have mentioned it in the past. I suppose that wouldn't work for a windows use like me 🙂

I don't think windows is officially supported (since it is not free), but I have seen some references to getting things build on windows, so it might even be possible.

What I am a bit annoyed with recently tbh is the wild explosion of different systems we recently started using elsewhere. IIRC, we now use uv, docker, podman and what not, besides poetry. Don't know if it's just me getting old, but it feels like we're just jumping onto every shiny new thing because it promises to solve some issues that we might have been perfectly able to solve with what we have, but now we depend on 5 new things. I suspect some of those decisions might have been influenced by AI, but that's just a feeling. Don't get me wrong though, I'm not at all saying our Guix falls under that!

Ah well, we reinvented the wheel probably a thousand times, and the wheel keeps getting better.

A technique I used during my PhD is first answer "Where am I going to write about this in my thesis?" before performing any action. Do you know where this ought to go in your thesis?

Kieran did encourage me in the past to write something about what he likes to call "PhD-ware". As in, principles of software development and lessons learned, in the context of Astronomy. I like that idea, especially given how much crap I've seen over the year, just at our institute alone. Given time constraints, this will probably just become an appendix to the thesis for now, but I might later publish it somewhere.

Good!

That said, without pointing fingers now, I did very recently encounter the very person who talked about all of this doing some very questionable commits somewhere. Pretty sure it was AI, just from the phrasing of things there. Oh and we almost ended up with en and em dashes in a YAML file, I wonder how they're intentionally typing those on a German keyboard 😶

We need to experiment to see how clankers can best fit in our workflow.

W.r.t. this issue, what the clankers seem to be good at is "Hey, this thing worked last month, now it doesn't, what is going on?" Something like that can be useful; maybe even proactively.