The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package^*

Major versions of the core package—that is, versions that add and/or modify functionality—are released approximately every 6 months, and are then maintained with releases that fix issues until the next major version is released. However, every 2 years a major release is designated as a long-term support (LTS) release, which continues to be maintained for at least 2 years Tollerud (2013). The motivation for LTS releases is to provide longer-term stable versions of astropy that users who require a high level of stability can make use of if they do not always need the latest features (this includes but is not limited to telescope operations and data reduction pipelines). The v5.0 release of the core package, replacing v4.0, was designated as LTS at the end of 2021; it will be maintained until the end of 2023.

The astropy core package is mature and stable, in that many features have been part of the core package for years with purposefully few changes to the software interface. This maturity and stability allow for the broader astronomy Python community to rely upon the astropy core package and build specialized packages within the Astropy ecosystem. Even many non-astronomical Python libraries have come to rely on astropy. The relationship between Astropy and the broader Python community is reciprocal, with the evolving needs of the community and the maturation of the broader scientific Python ecosystem driving much of the development of the astropy core package.

Within astropy, development may be roughly split into several categories: new features, intra- and inter-package interoperability, improvements to numerical precision, accuracy, and reproducibility, performance and documentation enhancements, and bug fixes. We discuss some of these in the subsections below.

Additionally, some features have been moved from the core package, either because they are of more general use outside of astropy, or because they are too specialized and belong in an Astropy-affiliated package. An example of the former is the copy of the IAU Standards Of Fundamental Astronomy (SOFA) software Collection ¹²⁵ Hohenkerk (2011) that Astropy carried, as well as the associated Python wrappers. Since this is basic infrastructure, with a release cycle set by SOFA, it made more sense to create separate packages that serve as astropy dependencies: ERFA Tollerud et al. (2021) and PyERFA (van Kerkwijk et al. 2021b). An example of something that was too specialized was a Virtual Observatory subpackage; the Simple Application Message Protocol (astropy.samp) is kept in the core package for now, but other parts more properly belonged in astroquery (see Section 4.3).

2.2.1. New and Planned Features

2.2.2. Interoperability

An important focus of astropy has been to increase the numerical precision and accuracy of its functionality, while, where appropriate, making sure results obtained using older versions of astropy are reproducible. Within each section below, we include in our descriptions how old results may be reproduced. The primary means are with options in configuration files and settings on runtime configuration objects called ScienceState. Notable improvements to astropy have been made to astropy.time, astropy.constants, astropy.coordinates, and astropy.cosmology, and we describe each in turn.

Time—In astronomy, time accuracy down to the (nano)second is frequently important. Whether for planning observations or crunching pulsar or very long baseline interferometry (VLBI) data, missing seconds meaningfully impact results. In past versions of astropy, when using Time, leap second tables had to be manually updated. Now, astropy automatically updates internal time data files to ensure that the correct leap second adjustment is always used. For reproducibility with older code, LeapSeconds may also be manually applied.

Constants—Measurements of physical constants (and the units defined from them) improve over time. Periodically the standardized systems of units and constants are updated to reflect these improvements. For instance, in 2019 the SI system was redefined by Newell & Tiesinga (2019), with an accompanying update to the physical constants in SI/CODATA 2018. astropy.constants now defaults to using the SI/CODATA 2018 values, with the units in astropy.units based on these constants. Most of Astropy and affiliated packages build upon astropy.units, so this update affects the entire Astropy ecosystem.

For reproducibility, astropy allows the constants' (and therefore units') definitions to be rolled back to prior values, e.g., to SI/CODATA 2014. For work sensitive to the values of the fundamental constants, we recommend including an astropy configuration file (i.e., astropy.cfg) with the work, which specifies the set of constants used.

Cosmology—astropy.cosmology contains classes for representing cosmological models. Bundled with the classes are commonly used Cosmology realizations, e.g., best-fit measurements from WMAP Bennett et al. (2003) and planck Planck (Planck Collaboration 2006).

When these important missions publish new measurements, astropy.cosmology is updated to include these results as realizations of the appropriate class (generally FlatLambdaCDM). The models used by WMAP and Planck do not always exactly correspond to astropy.cosmology classes. For instance, the Planck 2018 results Planck Collaboration et al. (2020) include massive neutrinos in Ω_matter,0, while in FlatLambdaCDM this mass contribution is stored in a separate parameter. Consequently, while some parameter values appear different from the source paper, the Cosmology realizations correctly reproduce the cosmological models.

Since the publication of Astropy Collaboration et al. (2018), the following cosmology realizations have been added: WMAP First Year (Spergel et al. 2003); WMAP Three Year (Spergel et al. 2007); Planck 2015 (Planck Collaboration et al. 2016); and the Planck 2018 best-fit cosmological parameters (Planck Collaboration et al. 2020).

astropy provides a configurable default cosmology, which is used in calculations done in a cosmological context. The default Cosmology has been updated to the Planck 2018 parameters. For reproducibility, this cosmology may be set to old defaults, such as the Wilkinson Microwave Anisotropy Probe (WMAP) Three Year values. The default cosmology is dynamically configurable using ScienceState, meaning a set of calculations may be run using different assumed cosmologies, and results compared between the two.

Coordinates—Being able to describe the coordinates of objects is fundamental across many knowledge domains. astropy.coordinates allows one to work with low-level positional (and velocity data all the way to high-level coordinate objects with reference frames, atmospheric information, etc. A central feature of astropy.coordinates is the ability to transform data between reference frames, e.g., ICRS (Arias et al. 1997) to GalacticLSR (Schönrich et al. 2010). However, there have been some historical limitations for spatially proximate transformations, impacting the usefulness of astropy.coordinates for ground-based telescopes and astrometry within the solar system. Transformations in the AltAz frame were reasonably precise for very distant objects, but wrong by up to several arcseconds, e.g., the location of the moon. Now these transformations are much more precise, down to the milliarcsecond level. Similar precision improvements were made to the Hour Angle-decl. frame transformations. Additionally, Ecliptic frames and associated transformations have been updated to correctly reflect the "true" and "mean" terminology.

For Galactic astronomers, the Galactocentric frame defaults have been updated to include more recent measurements (tabulated in the frame_defaults.references attribute). For reproducibility, old definitions are still available.

In some cases, feature enhancements and corner-case handling were added at the cost of performance. Hence, GitHub pull requests that improve performance are welcomed, and were even the main goal for the v3.1 release. For most subpackages, performance was improved in the Python code, but for some, the most time-critical pieces were rewritten in C and wrapped with Cython (Behnel et al. 2011; e.g., for convolution of images, sigma clipping, and converting time strings to binary). Furthermore, a particular effort was made to ensure astropy is thread-safe, so that it can be used on supercomputing clusters.

Performance in astropy is addressed based on feedback and requests, and through contributions from community members: if you find that parts of astropy are not performant, we encourage you to open a GitHub issue to start a discussion with the astropy core maintainers.

As part of the process of developing astropy into a long-term sustainable product, and to improve transparency and accountability, the Project agreed to write down and formalize our governance structure (partly supported by explicit funding for this purpose; see Section 3.4). At the 2019 Astropy Coordination Meeting, input was gathered from participants on what governance structures existed in the associated Open Source Software communities, and what would fit well with the needs of Astropy. This led to a "retreat" planned for March 2020, but due to the COVID-19 pandemic, this became a series of virtual meetings of the "Astropy Governance Working Group." This group drafted the APE 0 document (Aldcroft et al. 2021), which was eventually ratified and implemented by the "Astropy Governance Implementation Working Group" in Fall 2021. While the process emphasized flexibility and adaptability, it is the expectation that Astropy's governance will operate in this framework for at least the medium-term future.

The APE 0 (Aldcroft et al. 2021) document lays out the principles of this governance structure, so we refer the reader to that document for a more thorough description, and here we only highlight some key elements. The APE 0-based governance formalizes the building blocks that are already de facto true or previously discussed. With this in mind, it highlights the developer and user community of the Astropy Project as the ultimate sources of authority, as well as the core principle of "do-ocracy": those who do work for the Project (coding, training, or other harder-to-quantify contributions) gain more influence on the outputs of the Project by virtue of their effort. However, APE 0 adds the concept of voting members: a self-governed part of that community who are entrusted to elect the Coordination Committee (CoCo). While the CoCo predates APE 0, membership, rights, and responsibilities are now formally outlined. This role is mainly to facilitate consensus and make decisions when other mechanisms have failed, and includes powers that either require central authority or secrets (e.g., passwords). However, APE 0 also charges the Committee to devolve responsibilities and seek community input on these items as often as possible.

The first CoCo election took place in Fall 2021, electing a mix of prior and new members. New roles and voting members were also established. This suggests the process is already working to serve the long-term interests of the committee to both spread the coordination effort, and to ensure it is not dominated by the same people for as long as the Project continues. While other, more fine-grained governance improvements are planned for the future, it is clear the foundation is now in place.

As described in the introduction above, the Astropy Project is broader than the core astropy package as it also represents a number of core infrastructure, coordinated, and affiliated packages, educational initiatives, and user communities. In total, well over 1500 people have contributed in some way to the Project, whether by contributing code or documentation, reporting issues on GitHub, providing infrastructure support, writing tutorials, or helping to coordinate Astropy workshops. The vast majority of these contributors and participants have interacted as volunteers choosing to engage with the open-source and open-development software community. However, while the total number of contributors is large, the number of contributions and bulk of the maintenance of the Astropy Project components has primarily been sustained by a small number of heavily engaged or explicitly employed contributors (e.g., Astropy Collaboration et al. 2018), as is the case in many large, open-source software communities (e.g., Harris et al. 2020).

As an example of this within the Astropy Project, here we focus on contributions made to the astropy core package. Figure 2 shows the number of contributors who have contributed a given fraction of the total number of Git commits, separated into three separate time spans (eras) in the history of the Project. ¹²⁸ The annotations show that, within the early (2011 July–2015 January), mid (2015 January–2018 June), and recent (2018 June–2021 November) eras, 90% of the commits to the astropy core package have come from 18, 34, and 44 contributors. While the trend is encouraging—a larger pool of contributors are contributing more, fractionally—this emphasizes that, still, the vast majority of the work in astropy is done by a small subset (<20%) of the 115, 216, and 234 contributors, respectively, within each era.

Zoom In Zoom Out Reset image size

Since its inception, the most active contributors to the astropy package have spanned a range of career stages and paths. For example, Figure 3 shows a breakdown of the career stage/path at the midpoint of the three time eras defined in the previous paragraph for all contributors with more than 50 commits to the astropy package within each era. Over the ∼6 years between the early and recent eras, many contributors moved between these categories (i.e., have graduated as students to become research staff/faculty or software engineers), but also a number of early contributors moved on to other positions and stopped contributing to the core astropy package. Given this, it is encouraging that the numbers in each career stage/path have stayed relatively constant over the history of Astropy (with some showing growth), as new, active contributors have generally joined and stepped in to fill top-contributor roles within the Project. On the other hand, the total number of top contributors has not grown substantially, reflecting one of our principal challenges: sustaining a pipeline from users to contributors to maintainers of the package. One other thing to note in this context is that in all eras, 50%–60% of the software engineers who contributed to astropy were (or are) employed by the Space Telescope Science Institute (STScI), whose leadership has consistently provided resources (financial and personnel) toward sustaining Astropy.

Zoom In Zoom Out Reset image size

A final metric we consider as a diagnostic of the contributor base and contributions to the astropy core package is the rate of contributions and contributor participation, defined again using Git commits to quantify contributions. Figure 4 shows the monthly rates of commits and unique committers to the astropy core package over the history of the package. After an initial ramp-up in the commit rate and number of committers (lasting roughly from 2011–2014), the commit rate and number of unique committers have both remained fairly constant.

Zoom In Zoom Out Reset image size

With support from the Moore Foundation, the Astropy Project was allocated funding to support mentoring programs. In 2020 a call was made to submit proposals for Inclusion, Diversity, and Empowerment (IDE) initiatives for project-wide consideration on GitHub . This process was deemed the most open because it allowed for community-wide feedback to focus and improve proposal initiatives. Two highlighted programs that were selected and implemented via this process are described below.

Women of Color Code (WoCCode) is a peer-mentoring network for coders from traditionally marginalized groups, most notably women of color. Participants were invited to the WoCCode Slack space and encouraged to attend monthly webinars to share skills in the context of open-source software libraries. Every other month, a guest speaker was invited to talk about their career path and share a skill. Program participants were solicited in the fall of 2020, yielding 73 applications from 17 countries (48% from the United States, 37% from Africa, and 15% from the remaining continents). We selected participants who we identified as having a high potential for contributing to open-source projects: intermediate to advanced programming skills with a vocal interest in contributing. Of the 30 applicants invited to the program, 19 joined the community on Slack. Participants were organized into cohorts based on interest and each cohort was assigned one of three mentors that were also selected via open application. Mentors acted as a general resource to participants, gave one webinar, and organized a hack day. WoCCode also supported the registration of two participants to attend the American Astronomical Society Virtual Summer 2021 Meeting.

Participants rated the impact of the program as very high. In the final webinar, participants reported a change in their perception of coding in general, for example, accepting coding as something they can do for fun. Participants reported generally feeling comfortable asking questions and interacting with a community where "[e]veryone's thoughts are welcomed, no one is made to feel less important" and where one can enjoy "warm interactions with like-minded people." WoCCode is continuing into 2022 by broadening participation in the Slack space and publicly advertising guest webinar events.

In addition to the supported mentoring initiatives, the Astropy Project as a whole has taken steps to examine the representation of marginalized groups within the Project and search for avenues of improvement.

Astropy Representation at National Diversity Conferences: Several members of the Astropy community attended virtual conferences of the National Society of Black Physicists (NSBP) and SACNAS (focusing on Hispanic and Native American scientists across all STEM fields). Astropy representatives noted that one underlying topic came up multiple times: a major barrier for persons who come from underrepresented communities is the lack of pedagogical resources and expertise. For instance, software engineering skills and best practices are not part of the standard physics and astrophysics curriculum. Additionally, the larger astronomical community is still in the process of transitioning toward Python -based tools, hindering students not currently under advisement by someone with extensive Python expertise from getting involved with the open-source Python community. Projects such as Learn Astropy (see Section 4.5) could have a profound impact by empowering underrepresented groups because it provides a free, searchable, and accessible introduction to Python tools for astrophysical research. Representatives also noted that offering training and teaching materials to PIs of Research Experience for Undergraduate (REU) programs throughout the United States could be helpful for encouraging advisors to teach students how to use Astropy, create their own libraries, and use version control on open-source platforms like GitHub . Such materials could also be offered as a workshop at national diversity conferences themselves, as SACNAS solicits special session proposals each year.

With the goal of establishing a sustainable financial model for the Astropy Project, the CoCo, under the new governance charter (see Section 3.1), established a standing Finance Committee. This committee oversees the planning and allocation of finances on behalf of the Project. The Astropy Project accepts funds from institutions as well as individuals through NumFOCUS. ¹²⁹ Previously, no direct financial support was available for project development, and NumFOCUS covered most of the incurred operational costs. However, transitioning to long-term financial stability meant having an influx of funds in the form of institutional support, and hence the Astropy Project applied for and successfully acquired four grants over the last several years. These grants are administered by NumFOCUS on behalf of the Project, providing a neutral space not tied to any particular astronomy institution. The two major grants are briefly summarized below.

• Moore: The Gordon and Betty Moore Foundation awarded the Astropy Project a ∼$900k (US) grant in 2019 for the proposal "Sustaining and Growing the Astropy Project." Over a 3-year term, this grant supported the sustenance of the Project by providing funds for (1) the development and maintenance of astropy and its infrastructure, provided to existing project members and targeted hires, (2) the transition of long-term users to contributors and maintainers through mentorship efforts, (3) the formalization of a governance structure (see Section 3.1), and (4) the improvement of equity, diversity, and inclusion efforts in the Project (see Section 3.3). Some of this money supported travel for meetings, conferences, and workshops. This generous grant helped reduce reliance on volunteer-driven maintenance of the openly developed package and paved the way to acquire funding from US federal agencies, representing a major milestone for the success of the Project.
• NASA: A successful proposal to the NASA ROSES-2020 call, ¹³⁰ section E.7 (Support for Open Source Tools, Frameworks, and Libraries), granted the Astropy Project ∼$600k (US) to support its work. Funded work for a term of 3 years includes (1) project-wide infrastructure maintenance and improvements, (2) targeted work on areas of the Astropy core and coordinated packages, (3) enhancements to the Learn Astropy ecosystem and educational materials (see Section 4.5), and (4) support for the Astropy-affiliated packages. These goals support the sustenance of the Project by encouraging further development of the package ecosystem and community engagement.

In addition to the two grants described above, the Project benefited from a Gemini Observatory contract under its Science User Support Department, which awarded funds for development work that supports both the Astropy and DRAGONS ¹³¹ projects. The Project also received financial support from the Dunlap Seed Funding program at the University of Toronto to develop educational resources for scientific software packaging within the Learn Astropy framework. Moreover, the Project indirectly benefited from the NRAO ALMA Development Study program, which supported the grant "Linking CASA to the astropy ecosystem" ¹³² . All of these successful funding proposals show that the Astropy Project is vital for the astronomical infrastructure and community at large, and strengthen its promising and sustainable future.

The Project follows a set of principles to fairly and effectively distribute the above-described funds. By drawing inspiration from Numpy's NEP 48 ¹³³ , the APE 19 (Cruz et al. 2022) document lays out these principles. We refer the reader to that document for more details.

The Astropy Project supports the broader ecosystem by providing preconfigured infrastructure packages that the community can use to support and maintain their own software package infrastructure. These include tools and extensions that enable generating documentation and setting up automated testing, as well as providing package scaffolding for new projects.

Sphinx is a common and useful tool for generating documentation for Python packages. The Astropy Project maintains a default Sphinx configuration along with Astropy-specific extensions that can be easily added to community projects via the sphinx-astropy meta-package. This tool provides a preconfigured Sphinx setup compatible with Astropy projects, which includes several extensions useful for generating API documentation (sphinx-automodapi), allowing for numpy docstring parsing (numpydoc), embedded image handling (sphinx-gallery; pillow), advanced documentation testing support (pytest-doctestplus), and providing a custom documentation theme ideal for analysis packages (astropy-sphinx-theme).

Community package testing infrastructure is supported through the pytest-astropy meta-package, providing a unified testing framework with useful extensions compatible with both Astropy- and non-Astropy-affiliated community packages. This meta-package pulls in several pytest plugins to help with custom test headers (pytest-astropy-header), accessing remotely hosted data files in tests (pytest-remotedata), interoperability with documentation (pytest-doctestplus), dangling file handle checking (pytest-openfiles), data array comparison support in tests (pytest-arraydiff), subpackage command-line testing support (pytest-filter-subpackage), improved mock object testing (pytest-mock), test coverage reports and measurements (pytest-cov), and configuring packages for property-based testing (hypothesis).

The Astropy Package Template helps facilitate the setup and creation of new Python packages leveraging the Astropy ecosystem. This tool utilizes the Cookiecutter project to walk users through the process of creating new packages complete with documentation and testing support. Additionally, the package template generation process includes the ability to set up interoperability with GitHub, allowing for easy repository access from documentation, as well as an example GitHub Actions workflow to demonstrate the use of GitHub 's continuous integration tooling.

As defined in Astropy Collaboration et al. (2018), Astropy affiliated packages are astronomy-related Python packages that provide functionality that builds upon and extends the astropy core package that have requested to be included as part of the Astropy Project community. These packages support the goals and vision of Astropy of improving code reuse, interoperability, and embracing good coding practices such as testing and thorough documentation, but are not solely developed and maintained by members of the Astropy Project team. However, a subset of the affiliated packages—the "Astropy Coordinated Packages"—are maintained by the Project. All Astropy-coordinated and other affiliated packages are listed with detailed information on the Astropy website; a table summarizing key aspects of the affiliated packages is included in the Appendix below (Table 1).

Since the publication of Astropy Collaboration et al. (2018), there has been an expansion of affiliated packages for gravitational astrophysics, including PyCBC (Nitz et al. 2022) for exploring gravitational wave signals, lenstronomy (Birrer et al. 2021) for modeling strong gravitational lenses, ligo.skymap for visualizing gravitational wave probability maps, and EinsteinPy (Bapat et al. 2021) for general relativity and gravitational astronomy. There have also been several packages added to the ecosystem related to HEALPix: astropy-healpix for a BSD-licensed HEALPix implementation, and mocpy for Multi-Order Coverage maps. astroalign (Beroiz et al. 2020) has been introduced for astrometric registration, and python-cpl products (Streicher & Weilbacher 2012) has been added for ESO pipelines and VLT data. For ground-based astronomy, baseband (van Kerkwijk et al. 2021a) has added IO capabilities for VLBI, and SpectraPy (Fumana 2021) brings slit spectroscopy to the Astropy ecosystem. Other new affiliated packages include agnpy (Nigro et al. 2022) for active galactic nuclei jets, statmorph (Rodriguez-Gomez et al. 2019) for fitting galactic morphological diagnostics, dust_extinction for modeling interstellar dust extinction, feets for extracting features from time series data, and corral (Cabral et al. 2017) for managing data-intensive parallel pipelines. casaformats-io provides infrastructure for reading data stored in CASA table formats. saba gives an interface to the Sherpa (Doe et al. 2007) fitting routines, BayesicFitting (Kester & Mueller 2022) provides an interface for generic Bayesian inference, and sbpy (Mommert 2019) enables calculations for asteroid and cometary astrophysics. Finally, synphot (STScI Development Team 2018) provides an interface for synthetic photometry.

Several of the coordinated and affiliated packages described in Astropy Collaboration et al. (2018) have had substantial improvements. astroquery (Ginsburg et al. 2019) has added access to roughly a dozen new missions and data services, including the JWST archive, and the Project has switched to a continuous release model: every time a change is committed to the main development branch it is published on the Python Package Index (PyPI) and available for installation. (However, formal releases are still done a few times per year.) photutils (Bradley et al. 2022a) released its first stable version, indicating that the API will change less frequently, and there have been several significant performance improvements. ccdproc (Craig et al. 2015) released a new major version, bringing better performance to some image combination operations. The regions package (Bradley et al. 2022b), for manipulating ds9-, CASA-, or FITS-style region definitions (Joye & Mandel 2003), added new ways to manipulate regions, introduced new region types and obtained expanded visualization and interactive region editing capabilities. reproject's (Robitaille 2018) major new feature is a function to align and co-add images to create a mosaic; better support for parallelization was also added. specutils, the package that defines containers for 1D and 2D spectra (Earl et al. 2021), had its first stable release and the addition of classes to read JWST data. stingray (Huppenkothen et al. 2019) has had a major performance overhaul. The package gammapy (Deil et al. 2017) has had major performance improvements and has unified its API in preparation for its first stable release.

astroquery (Ginsburg et al. 2019) is the Astropy-coordinated package for interacting with online archives of astronomical and related data. It contains over 50 modules for querying astronomical databases, large and small. In particular, since the publication of Astropy Collaboration et al. (2018), significant contributions to astroquery have come from several of the major archives, including the European Space Agency (ESA), the Mikulski Archive for Space Telescopes (MAST) at STScI, the Infrared Science Archive at the NASA Infrared Processing and Analysis Center (IRSA at NASA IPAC), the Canadian Astronomical Data Center (CADC), and the Atacama Large Millimeter/Submillimeter Array archive (ALMA). These contributions represent a formal acknowledgment of the utility of a centralized tool suite for archive interaction from Python . The widespread usage of astroquery is apparent from the range of keywords represented in the journal articles that cite the astroquery paper Ginsburg et al. (2019); everything from asteroids to galaxies is represented. Additionally, astroquery functionality has been built into several special-purpose Python packages, such as LightKurve (Lightkurve Collaboration et al. 2018) and SORA (Gomes-Júnior et al. 2022). More than 4,000 repositories on GitHub make use of astroquery in some way, and astroquery features in a large number of tutorials on various facets of astronomical analysis.

Many of the existing and newly contributed tools rely on virtual observatory (VO) tools. These use the underlying package pyvo, which has also recently become an Astropy-coordinated package. While many or all of the functions provided in astroquery can be achieved through direct use of VO tools implemented in pyvo, the astroquery interfaces more closely resemble the web interfaces that are more familiar to most users.

In this section, we highlight a few efforts in observatory- or mission-driven development that have contributed to Astropy, and vice versa.

4.4.1. JWST

4.4.2. Cherenkov Telescope Array

4.4.3. Data Central and the Anglo-Australian Telescope Archive

Learn Astropy—is an umbrella term that acknowledges the broad educational efforts made by the Astropy Project, which is led by the Learn Astropy Team. The efforts focus on developing online content and workshops covering astronomy-specific coding tasks in Python . As introduced in Astropy Collaboration et al. (2018), there are four different types of Learn Astropy content: tutorials, consisting of Jupyter Notebook lessons that are published in HTML format online; guides, which are a series of lessons providing a foundational resource for performing certain types of astronomical analyses; examples, which are snippets of code that showcase a short task that can be performed with astropy or an affiliated package; and documentation, which contains more detailed information about the code base and user interface. This categorization drives content development, infrastructure choices, and the appearance of the Learn Astropy website. The Learn Team meets weekly to work on creating, expanding, and improving these educational resources.

The Learn Team recently relaunched the main website and search interface for Learn Astropy in 2021 with a new infrastructure platform, built around full-text search and interactive filtering functionality, with the goal of making content more easily searchable and discoverable as the Learn Astropy content catalog expands. This work has been supported in part by a grant from the Dunlap Institute. We have adopted Algolia, a search-as-a-service cloud platform, to store the full-text and metadata records of Learn Astropy's content. The new Learn Astropy website is a JavaScript (Gatsby/React) application that uses the Algolia service to power its search and filtering user interface. Our Python -based application, Learn Astropy Librarian, populates data into the Algolia service. We tuned the Librarian around specific content formats (such as Jupyter Notebook-based tutorial pages and Jupyter Book-based guides) to more accurately index content and heuristically extract metadata. A consequence of the new platform is that we now maintain and compile content separately from the website application itself, enabling new content types. Tutorials, which are written as Jupyter Notebooks, are now compiled into their own Learn Astropy website using nbcollection. Guides, which utilize the Jupyter Book build infrastructure, are also deployed as separate websites using GitHub pages. This architecture opens future possibilities of indexing third-party content, hosted elsewhere, such as on institutional websites.

We currently host 19 tutorials written as interactive Jupyter Notebooks and rendered into static HTML pages with the infrastructure described above. The tutorials span a range of astronomical topics, from general tasks like reading or creating FITS files with different content or working with astronomical coordinate systems, to more specific exercises like analyzing spectroscopic data from the UVES instrument. Now that the backend infrastructure is stable, we are interested in collecting new content to serve and share. Ideas for contributions, new tutorials, or suggestions for new tutorial series to be made searchable through the Learn Astropy interface can be raised by creating a GitHub issue in the Astropy Tutorials GitHub repository. We currently host one Guide—a longer-form walkthrough of a more complex concept or topic—that is focused on CCD image reduction and photometry, but are also actively seeking new material that would be suitable for new guides.

Beyond developing and serving educational content, the Astropy Project has been conducting workshops at winter meetings of the American Astronomical Society since AAS Meeting 225 in 2015 January. Up to the start of the coronavirus pandemic, these were full-day in-person workshops with as many as 90 participants and a dozen facilitators from the Project. During the pandemic, these workshops were moved to an online format and split into basic and advanced sessions. Additionally, beginning with the AAS 238 online meeting, the workshops have been expanded to summer AAS meetings. The Learn team finds that the workshop audience is best found at AAS meetings as opposed to more general Python meetings, as the content tends to be more applicable for students and researchers in astronomy and astrophysics.

The Astropy Project recently provided another mode of community engagement at AAS Meetings 235 and 237 by organizing a NumFOCUS Sponsored Projects booth in the AAS Exhibit Hall. Funding for the exhibit hall was provided alternately by NumFOCUS and later by the Moore Foundation funding. The booth hosted a series of Q&A special sessions during AAS Meeting 235 and webinars during the virtual AAS 237 meetings, to provide the general astronomy community information and access to experts on a variety of open-source astronomical tools.

The focus of Learn Astropy over the coming months and years is to further improve the interface for sharing educational content on the main Learn Astropy website, and to facilitate the development of new content that highlights astropy and Astropy-affiliated package functionality. However, we simultaneously also plan to solicit indexing and ingestion of third-party tutorial series, to provide a unified interface for identifying astronomy-specific educational content that demonstrates the functionality available in the vast ecosystem of open-source software packages. We also plan to look for opportunities to expand the reach of Astropy workshops beyond the AAS meetings.

The Astropy Roadmap is a document in the astropy-project GitHub repository within the Astropy organization that captures high-level actionable items that the Astropy Project aims to undertake to improve the health and stability of the Project. It is a static document that is revisited regularly at the Astropy coordination meetings, intended to keep track of progress and write new versions as needed. There is a related project board linked to the Roadmap document that holds specific issues and efforts related to Roadmap items. The project board is a living document that is continually updated as work is planned, assigned, and completed.

All items in the Roadmap have been agreed to be priorities for the Astropy Project, and are color coded based on resources (both time/effort and developers/expertise) needed to complete the item. Green items are well underway, have sufficient resources/support, and a plan in place for completion; for example, efforts to improve the discoverability of documentation and educational materials by overhauling the Learn website, which is underway (see Section 4.5). Orange items are well defined, and work for acquiring sufficient resources underway; for example, the goal of providing next-generation spectroscopic reduction, analysis, and visualization tools usable by individual researchers and larger surveys. Red items do not yet have a plan for implementation and need more resources; for example, the goal of improving interoperability with performant I/O file formats and libraries such as HDF5 and Dask. The Astropy Roadmap originates from the 2021 March Astropy Coordination meeting where it was first drafted before being handed off to a newly formed Astropy Roadmap working group for completion. The Roadmap in its current form was adopted via pull request in 2021 December.

Over the history of Astropy, the needs and challenges of the Astropy Project have evolved, but many common themes have persisted and are shared by other open-source software communities. However, Astropy also faces a number of unique challenges stemming from the fact that the Project is in a special position in the astronomy software stack: the Astropy code and ecosystem are more specialized than core scientific tools like numpy and scipy, but more general than software packages developed to enable research in specific subdomains such as exoplanet (Foreman-Mackey et al. 2021). Below we discuss a few challenges faced by the Project in more detail, but we stress that this list is not complete or exhaustive.

• 1.  
  Supporting an active group of maintainers: Throughout its history, the development and maintenance of code and educational materials in the Astropy Project—the astropy core package, coordinated packages, Learn Astropy repositories, and Project infrastructure packages—has been done by a mix of volunteer effort and institutional support. At present, the named roles and maintainer positions in the Astropy Project are also split roughly evenly between these types of contributors. As is the case with many other open-source software communities, the culture of the Project and overall coordination relies heavily on volunteer effort. We therefore have significant amounts of critical infrastructure or core functionality that is either maintained with no redundancy or unmaintained as contributors naturally move on from and stop contributing to the Project. A prime example within the core package is the astropy.fits subpackage, which is largely in maintenance mode and maintained by just one contributor (despite the significant code base and user community behind this subpackage). While we now have funding to support the longer-term sustainability of the Project and thus to address these issues, we are finding it difficult to use this funding in ways that address our needs. For one, many of the needs of the Project require long-term and sustained funding rather than short-term contracts. Furthermore, the specialized nature of the software requires being able to hire or contract software engineers with domain knowledge about astronomical software tools. Finally, we want to be able to preserve and sustain the engaged community of volunteers, software-engaged researchers, and existing software engineers that contribute to the Project and that have enabled its success in ways that support their careers. These challenges will require both cultural and larger systemic changes to astronomical research and career paths, and so will take time to influence and implement.
• 2.  
  Diversity of users, contributors, maintainers, and coordinators: The Astropy Project aims to be made up of members at all levels (users to contributors to coordinators) with a diverse set of skills, backgrounds, and experiences. While Astropy is used globally—for example, Figure 5 shows the number of users per country who accessed the astropy core package documentation in 2021— our user community is underrepresented in non-English-speaking or -teaching countries, and our maintainers and coordinators are fairly narrowly concentrated in the United States and Europe (see the circle markers on Figure 5). The gap in users in many countries is understandable given that much of our content and documentation is written in English and therefore only available in other languages through imperfect, automated translation services (like Google translate). The fact that our maintainers and coordinators are generally in the US (even then, mostly on the East Coast) and Europe is a reflection of where institutions have explicitly supported Astropy, and where volunteers have had the ability to contribute to the Project.
• 3.  
  Long-term and sustained funding for maintaining infrastructure: The software and packaging infrastructure within the Astropy Project includes (but is not limited to) Continuous Integration (CI), test plugins, documentation builds, and package delivery. Keeping the infrastructure working is essential in software development and operations. This kind of work has low visibility and is not attractive to scientists or volunteer contributors; therefore, it is most effective to delegate such tasks to paid roles. Astropy secured a 3-year grant for this but the situation remains uncertain once the grant runs out, as infrastructure evolves and requires constant maintenance and upgrades. Some of the engineers currently in these roles are also employees of research institutions that may leave the Project without much notice, adding to the uncertainty factor.
• 4.  
  Understanding, communicating with, and engaging our user communities: The Astropy Project is used by a number of intersecting and overlapping communities, such as telescope operations and data management, data archives, researchers, students, educators, and the general public. These communities have different needs from and uses for Astropy, many of which we are only aware of through individual engagement from members of these respective communities. This kind of communication may provide biased perspectives on these communities. We therefore need to develop more formal and democratic ways of engaging these communities, such as through running and analyzing user surveys, or creating and engaging with user groups that can better represent these communities. Additionally, as the Astropy ecosystem has and continues to grow, we need to ensure that users in these different communities have access to educational materials to train new users to use Astropy and Python software tools. This may involve tailoring workshops, tutorials, and educational materials to these communities; however, this represents a significant maintenance and content development burden on the Project.

Zoom In Zoom Out Reset image size