Page tree
Skip to end of metadata
Go to start of metadata


This is intended as a list of survival tricks for users who are particularly reluctant to using VO tools, but need to retrieve data from VO services in VESPA.

1- Querying data services

It is easy to see the content of a data service in VESPA.

In the VESPA query interface, just click the "Submit" button. This will display a list of connected services with the number of entries in each service.

If you know which service you want to look at, click on the leftmost icon (table-shaped) on its line. This will display the content of the service, paginated. You can change the number of entries per page, or navigate through pages. You can then select your data visually, possibly with the help of thumbnails.

Going back to the query page, you'll find it easy to simply add a filter on target_name, target_class, or dataproduct_type parameters, or even on time_min / time_max.

• It may be useful to stress that, independently from VO access, EPN-TAP provides a convenient way to describe personal datasets with a standard set of keywords (axes and physical quantities).

2- Accessing data

Most data services provide links to data files. In EPN-TAP services, such links are available under the access_url parameter. To retrieve several files at once, you only need to click on selected lines and use the "Data Selection > Download" item from the menu on the bottom left; this will download a zip file.
In other cases, the data are included in the service table directly, together with the descriptive parameters.
Some services propose several versions of the data files, e.g. an ascii file and a conversion to VOtable. Beware that in some cases the ascii version is the native format available in an external archive, while the VOtable version may have a different, updated or corrected content (this is the case of IKS or spectro_planets). In other cases, the conversion from ascii is performed on the fly and the content is identical (this is the case of M4ast or Titan).

2.1 Attached files

Many data services link to files in VOtable or fits format, which are two VO standards. These files can be downloaded easily on your disk through the VESPA portal or TAPhandle (the VESPA portal can also send them directly to the appropriate VO tools).

  • Reading fits files in your environment should be obvious in most cases - fits libraries are widely available (however, TOPCAT will only open tables, while ds9 will only open images adn cubes - other extensions may require specific processing).
  • A VOtable can be read and used in TOPCAT. If you prefer to use a more familiar environment, you can save them from TOPCAT in various formats, including simple ascii, fits, or csv which can be read in many software - or even html or latex for use in documents.
  • Several services provide a link to a web service under access_url, in particular for OGC/GIS services (e.g., CRISM, USGS_maps, M3…). Such links can be send from VESPA to QGIS through the EPN-TAP plug-in (jra-t4-tools-qgis-plugin).

2.2 Data included in the service table

The result of a query (as displayed in the VESPA portal) is also formatted as a VOTable, which can be read in TOPCAT and from there can be stored in ascii or other formats. The query can be issued directly from TOPCAT using its TAP interface (which is less user-friendly than the VESPA portal). Converting the complete result table will also provide access to the descriptive parameters.

VESPA's Google Sheet Add-on may also be handy in this case, see here: TAP Sheet (Google Sheets Add-On).

3- IDL

The most helpful IDL libraries in this context are NASA's Astron library, and possibly SSW.

The SSW (SolarSoftWare) library provides a complete VO interface, including queries to the registry and to individual services declared there (should work under Linux): http://www.lmsal.com/solarsoft/ssw_setup.html.
However, this library is not maintained and is partly obsolete, and may not work properly.

3.1 Fits files

Several fits routines are available in the Astron library: readfits is the simplest one to use, and handles many fits variations. In case of special formatting (e.g., multiple extensions), mrdfits may be required to access the data.

Corresponding routines are available to write fits files after modification: writefits and mwrfits (see their doc).

3.2 VOtable

Several routines or libraries handling VOtable exist, but they do not all comply with the standard.

The VESPA-provided read_vot routine reads all VOtable using the stilts library associated to TOPCAT (will work uner Linux and Mac, and with limitations under Windows): https://github.com/epn-vespa/IDL_VOtable

An equivalent write_vot routine exists but has limitations (e.g., it will not provide a fully informed FIELD description) - see routine headers for other uses (updates, etc).

The VOtable writer routine from the SSW library has been used with success to feed TOPCAT with data processed in IDL. This is particularly handy to write tables of spatially sparse observations, that TOPCAT can then integrate in Healpix maps much more efficiently than IDL (see here: 1- Various use cases on planetary surfaces with Aladin & TOPCAT).

3.3 PDS files

Some services may provide links to PDS3 files (e.g., VVEX). Those are read under IDL with the VESPA-provided library LecturePDS (https://github.com/epn-vespa/LecturePDS), or with the PDS Small Bodies Node library readpds: https://pdssbn.astro.umd.edu/tools/tools_readPDS.shtml
No multipurpose library exist in other environments, however ad-hoc routines may be available in other languages for specific datasets.

The SBN also provides a routine to read PDS4 data under IDL (no such files currently exist in EPN-TAP services).

3.4 GIS/OGC formats

IDL has WMS and WCS interfaces which allow it to browse GIS servers referred to in EPN-TAP services (type ogc_wcs or ogc_wms in the command line).
However, a direct call of the file url is OK and much simpler, especially if you've already selected files from the VESPA interface:

; from the EPN-TAP table displayed in VESPA
access_url= "http://access.planetserver.eu:8080/rasdaman/ows?&SERVICE=WCS&VERSION=2.0.1&REQUEST=GetCoverage&COVERAGEID=hrl000052ff_07_if155l_trr3&FORMAT=image/tiff"
; will write file in working directory (see options to IDLnetURL Get method for anything more elaborated)
oUrl = OBJ_NEW('IDLnetUrl')
pipo=oUrl->Get(URL=access_url)
help, pipo
   PIPO            STRING    = '/YourPath/CRISM/idl.dat'
iopen, pipo, popo
help, popo
   POPO            FLOAT     = Array[438, 60, 28]
tvscl, popo(200, *,*) >0.25<0.55
OBJ_DESTROY, oUrl


3.5 Locating files inside an EPN-TAP service

Similarly, if you already know the name of the files of interest, use VESPA's file grabbing web services with the IDLnetURL object. Usage from IDL is described here: File grabbing interface

This type of access is intended for easy pipeline processing.

3.6 Access to the registry of services

The SSW library provides a complete VO interface under Linux, including queries to the registry and to individual services.
However, this library is partly obsolete and may not work properly.

4- Python

4.1 VOtable & fits files

Routines exist to handle these two formats in the astropy library (https://pyvo.readthedocs.io/en/latest).

4.2 PDS files

The PDS Small Bodies Node also provides routines to read PDS4 data under Python (no such files currently exist in EPN-TAP services): https://pdssbn.astro.umd.edu/tools/tools_readPDS.shtml

4.3 Locating files inside an EPN-TAP service

The file grabbing web services from VESPA (and their use from Python) are described here: File grabbing interface

This type of access is intended for easy pipeline processing.

4.4 Querying an EPN-TAP service via Python

Astropy + pyvo libraries cover some VO access.

A short tutorial is available here (from Baptiste Cecconi):
https://github.com/epn-vespa/tutorials/blob/master/pythonic-access-tutorial/pythonic-access-tutorial.md

An example of query from Jupyter notebook is provided here (from Baptiste Cecconi):

https://github.com/epn-vespa/vespamap17-hackathon/blob/master/vespa-mapping-jupyter-samp/VESPA-SAMP-CRISM-to-GeoPandas.ipynb

Direct query via pyvo (from Florence Henry):

In [1]: import pyvo
In [2]: service = pyvo.dal.TAPService("http://voparis-tap-planeto.obspm.fr/tap")
In [3]: query = "SELECT target_name, inclination, eccentricity FROM exoplanet.epn_core WHERE semi_major_axis < 5"
In [4]: results = service.search(query)
In [5]: results
Out[5]:
<Table masked=True length=2050>
…

In [6]: type(results)
Out[6]: pyvo.dal.tap.TAPResults

In [7]: len(results)
Out[7]: 2050

In [8]: results[0].get('target_name')
Out[8]: '11 Com b'

In [9]: results[2].get('target_name')
Out[9]: '14 And b'



  • No labels
Write a comment…