Welcome to ECODATA-Prepare!#

Overview#

ECODATA-Prepare is a set of Python-based apps to access and process remote sensing and other environmental data products and prepare them for further use along with animal movement data. ECODATA-Prepare is designed to be used along with the ECODATA-Animate tool to create customized movies of animal movement data, and can also be used to prepare input to resource, habitat and step selection models. Development is supported by MathWorks® and the NASA Earth Science Division, Ecological Forecasting Program, as part of Room to Roam: Y2Y Wildlife Movements project.

Within ECODATA-Prepare,

  • The Tracks Explorer App visualizes movement track location points and point densities with several static maps, allows selecting a spatial frame around the track points that is relevant for analysis, and prepares a .geojson file that can be used in the NASA EARTHDATA AppEEARS interface to extract many different remote sensing data products within this research area. This app also provides the longitude-latitude coordinates of the frame around the research area, which can be used to request data from ECMWF or many other environmental data sources, which can be converted to NetCDF format if needed for further steps described below.

  • The Gridded Data Explorer App allows you to interpolate and subset the temporal and spatial resolutions of environmental data in the form of a temporal stack (or single static map) NetCDF file. It can also read in several other data formats (which could be resaved as NetCDF). You can also read GIS polygons as shapefiles and mask the environmental data outside or inside the polygons. It subsequently calculates data summaries by period and polygon.

  • The Subsetter App allows you to clip relevant features out of large GIS files.

  • The Movie Maker App produces an animation file out of a stack of static maps which were produced by the ECODATA-Animate program.

Read our overview of how to use Movebank, MoveApps, ECODATA-Animate and ECODATA-Prepare to discover and process tracking data, remote sensing data, shapefiles and other layers to create custom visualizations and input for ecological analysis. To try the software before working with your own data, see our example based on publicly-available data sources.

For help, submit a GitHub issue or contact support@movebank.org.

ecodata_workflow

Python package#

The functions underlying the ECODATA-Prepare apps can also be used directly as a python package (i.e. without the GUI inferface). The package documentation is here.

Contents#

Installing the apps#

Installers#

Download installers for the latest release:

OS

Architecture

Download

Windows

x86_64

ecodata-0.2.7-Windows-x86_64

macOS 14

arm64 (Apple Silicon)

ecodata-0.2.7-MacOSX-arm64.pkg

(*) macOS 13

x86_64

ecodata-0.2.7-MacOSX-x86_64.pkg

Linux

x86_64

ecodata-0.2.7-Linux-x86_64.sh

(*) Use this installer for all Macs that are not an Apple Silicon (M-series) Mac running macOS 14 (i.e., use for all x86 (Intel chip) Macs, or any Mac running macOS 13).

Relase notes for the latest release can be found here, along with all available installers. Command line (.sh) installers for Mac are also available. You can see all releases (including previous versions and pre-releases) here.

  1. Download the correct installer for your operating system.

  2. Run the installer and follow the prompts. You can change the default installation location if you wish. This installation location is where the program files will be installed (i.e., files not intended to be accessed by the user after installation). If you have a previous version already installed in the default location, you will need to remove the previous version before installing the new version, or install the new version in a different location. If opening the .pkg in Mac, you might get a message that it “cannot be opened because it is from an unidentified developer”. If this happens, you can right-click or press Control click on the application, and then select Open. You can then select Open to run the installer.

  3. A file used to launch the apps (ecodata.command for Mac and ecodata.bat for Windows) will be copied to your Downloads folder. You can move this file to a convenient location after the installer is finished running. Note: If you already have an ecodata.command or ecodata.bat file in your Downloads folder from a previous installation, remove it before installing the new version.

After the installation is complete, you can get started.

Alternative installation methods#

The apps can also be installed from the command line. If you have issues with the first installation method (e.g. your computer’s security settings are preventing the installer from running), you can use this method as a backup.

Updating ECODATA-Prepare from the terminal#

You can check which version of ecodata you have installed by running:

conda activate eco
conda list ecodata

You can see new releases on the releases page

If you want to install a new release, you can update your version of the apps by running the command below:

conda activate eco
conda update ecodata

After the installation is complete, you can get started.

User guide#

Tip

Want a downloadable version of the docs? This documentation is also available for download as a PDF, Epub, or zipped HTML.

Contents#

Getting started#

Overview#

ECODATA-Prepare is a set of four Python-based apps to read, process, and create animations from animal tracking data and gridded environmental data. Read here for an overview and to learn how to install, contribute and get support. Before using this program, gather files you want to use with the app. This could include:

  • a .csv file of tracking data, in Movebank’s .csv format

  • NetCDF files of environmental data (you can order these from NASA or ECMWF)

  • shapefiles

  • a folder of sequentially-numbered .png files that are frames for an animation (you can create these with ECODATA-Animate)

If needed, see the installation instructions to install or update the program.

See below for instructions to open the program. When it opens, it will display the main panel, showing the four apps. Click on an app to launch it. From there, you can navigate between apps within the interface, or switch between them by pasting these URLs in your browser window:

ecodata-prepare_panel

Opening ECODATA-Prepare#

Note

These instructions assume that you installed ECODATA using the Installers. If you installed ECODATA using one of the alternative methods, you will need to launch the apps using the instructions there.

Double-click on the launch file (ecodata.command for Mac, ecodata.bat for Windows). A terminal window will open, indicating that the apps are launching. When the apps finish launching, a window will open on your default web browser, showing the app gallery. The apps are running locally at localhost:5006. Note: There may be a short wait (10+ seconds) before the browser window opens, particularly if you are launching the apps for the first time.

You may receive a message “Do you want the application “python3.11” to accept incoming network connections?” You can click Allow.

Keep the terminal window open while running the app. To shut down the apps, close the terminal that opened when the apps launched.

Tracks Explorer#

tracks_explorer_map

App features#

With the Tracks Explorer App, you can

  • Visualize point locations from your Movebank data as individual points or aggregated point density. You can select from a variety of map tiles to use as a background.

  • Automatically generate a spatial frame around the track points and save this as a .geojson file that can be used in the NASA AppEEARS interface to extract remote sensing products for the research area.

Using the app#

tracks_explorer_start Data from Hebblewhite and Merrill, study Hebblewhite Alberta-BC Wolves

  1. If you haven’t already, download a local copy of the tracking data you want to use, in Movebank’s .csv format. It is possible to subset or combine data from different studies.

  2. Launch the Tracks Explorer App.

  3. Under “Select File”, paste the full filepath to a .csv file of tracking data, including the file name. In the current version, this .csv file needs to use Movebank’s format. An example filepath is “/Users/myname/MovebankData/wolves.csv”.

  4. Click “Load data”. An interactive map will appear of your dataset, where you can pan/zoom to explore the dataset. If a map does not load, view the status message. The message “Error.. Check options and try again” likely indicates that your filepath is incorrect.

  5. You can change the background map tile using the drop-down menu in the sidebar.

  6. Toggle the “Datashade tracks” checkbox in the sidebar to turn on/off the point density aggregation feature. When unchecked, individual semi-transparent points will be displayed. Note that for very large datasets, plotting will be slow if the “Datashade tracks” option is turned off.

  7. A boundary for the tracks dataset is shown on the map in red. The default option uses a rectangular boundary with a buffer size of 0.1 (scale relative to the extent of the tracks). You can adjust the buffer size or change the boundary shape to a convex hull using the widgets in the sidebar.

  8. To save the boundary to a .geojson file, scroll down and if needed, edit the file name and location under “Output file”. Then simply click the “Save extent” button.

Gridded Data Explorer#

gridded_data_explorer

App features#

With the Gridded Data Explorer App, you can

  • Read, interpolate and subset the temporal and spatial resolutions of environmental data in the form of a temporal stack (or single static map) NetCDF file.

  • Read GIS polygons and mask the environmental data outside or inside of the polygons.

  • Calculate data summaries by time period and spatial area.

  • Save processed environmental data and summary statistics for further use.

Using the app#

Tip

Throughout the App, scroll down to view status messages indicating application progress and next steps.

  1. Under “Input environmental dataset”, paste the full filepath to a NetCDF (.nc) file received from NASA AppEEARS (see instructions) or any other source. The filepath should include the file name. An example filepath is “/Users/myname/RS_Data/NDVI.nc”.

  2. Click “Load data”. If it is able to read the file, the app will load available variables from the file into dropdown lists for Latitude, Longitude, Time and Variable of interest, and will attempt to select the correct variables automatically. If nothing happens, you might need to correct the filepath.

  3. If the app does not assign the correct variable names from the file, you can specify them manually by clicking on the field and selecting from the dropdown list.

  4. In the “Variable of interest” dropdown, select the variable you would like to explore.

gridded_data_explorer_input

  1. Once the input parameters are set, click “Update variable names”. Plots of the data will appear.

  2. At the top of the area with the plots, there are two tabs (“Charts” and “Data”). If you click the “Data” tab, you can see more information about the file contents, including the dimensions of the dataset and the variables in the file. Click on the “Show/Hide attributes” icon to the right of a variable to view variable metadata including long names, units, and legend classes.

gridded_data_explorer_data

  1. In the “Charts” tab, there are a variety of options to explore the selected variable in space and time.

  2. Filter by area: Under “Input polygon file”, you can input a polygon file to select or mask data inside polygon boundaries. Selecting will keep data inside polygons, and masking will keep data outside polygons. Paste the full filepath to a polygon file (e.g., .shp), including the file name, and click “Load file”. When the file loads, it will be plotted on the Chart view above. Select and option and then click “Update filters”. To remove the filter, click “Revert filters”.

  3. Filter by time: To filter the data for specific times, use the options under “date range selection” under Selection Options on the sidebar. There are time filtering options in the sidebar if you want to filter the data for a specific time range or time conditions (e.g., daytime only, specific seasons or months). You can use the time slider bars to filter by a range of values or enter discrete values into the box. For example, to filter by month you would click the “Month” tab above the “Range Values” option and either use the sliding bar to restrict your data to months of interest or enter numeric months into the Discrete Values box (note that when you click in this box, a dropdown appears with the month values from your Appeears dataset). Available ranges and categories for discrete filtering will depend on the content of your dataset. Select your options and then click “Update filters”. If you want to remove the filters you have applied, click “Revert filters”.

  4. Resample by time: It is possible to resample the data by time. The app will estimate attribute values from the original values at the requested time schedule by interpolating when upscaling, or using means for downsampling. Upscaling could be used to show smooth transitions in an animation with daily timestamps of a product with a 16-day resolution; downscaling could be used to reduce storage and processing needs for a large hourly dataset if you need only daily data. Under “Processing Options” in the sidebar, select the desired time frequency and click “Resample time”. Once processing is complete, the status at the bottom of the page will show “Dataset Resampled”.

  5. Coarsen spatial resolution: It is possible to coarsen (downsample) the spatial resolution of the data using block aggregation, for example to reduce file sizes or processing time for subsequent uses. Under “Processing options” in the sidebar, select a window size for coarsening and click “Coarsen dataset”. The window size will indicate the number of pixels in the x and y directions over which a mean should be calculated. Once processing is complete, the status at the bottom of the page will show “Aggregation completed”.

  6. After applying filters or processing to the data, you can save the results as a new .nc file. Under “Output file”, choose a name for the file. You can specify a filepath, or provide only a file name to have it saved in a default folder on your computer. Select “Save dataset”. Below, a message “File saved to: [filepath]” will indicate where the file is stored.

  7. Calculate summary statistics: First select grouping options in the “Calculate statistics” panel, then click “Calculate statistics”.

gridded_data_explorer_data_stats_filters

Once processing is complete, the status at the bottom of the page will show “Calculations completed”, and a table of the summary statistics will appear in the “Statistics” panel. You can save this output as a new .csv file. Under “Output file for statistics”, choose a name for the file. You can specify a filepath, or provide only a file name to have it saved in a default folder on your computer. Select “Save statistics”. The status message will update to indicate where the file is stored.

gridded_data_explorer_data_stats_results

Subsetter#

subsetter

App features#

With the Subsetter App, you can

  • Clip relevant features out of large shapefiles based on a bounding box, the extent of a tracking dataset, or a custom polygon.

  • Save the extracted subset for further use.

Using the app#
  1. In the “GIS file” field, paste the full filepath to a GIS dataset (e.g., a shapefile or file geodatabase) from which you want to extract a subset. The filepath should include the file name, or if the dataset consists of multiple files (e.g., for a shapefile) the folder in which the files are saved. An example filepath is “/Users/myname/global_roads_dataset.gdb”.

  2. Next, define the spatial area for which you want to extract data from the GIS dataset. You can do this using a bounding box, track points, or a bounding geometry. There are specific settings for each option:

    • Bounding Box: Specify the latitude and longitude range for a rectangular bounding box. Provide coordinates in decimal degrees, WGS84 coordinate reference system.

    • Track Points: Provide the full filepath, including the filename, for a .csv file of tracking data containing location-long and location-lat in decimal degrees, WGS84 coordinate reference system. The app will draw a boundary around the track data: choose whether to do this using a rectangular or convex hull method, and optionally define a buffer around which to expand the boundary. This buffer is relative to the extent of the tracking data (the selected buffer size times the greater of the north-south or east-west extent).

    • Bounding Geometry: Provide the full filepath, including the filename, for a file (.geojson, .shp or .gdb) defining a polygon. This could include output of the Tracks Explorer App.

  3. For any of the options above, decide whether to clip features that overlap with the edge of the spatial area defined in the previous step. Uncheck “Clip features at boundary edge” to retain complete lines or polygons that overlap with the boundary edge. By default, the app will plot the data subset once it is created, along with the geometry that was used to create the subsetting boundary, and the actual geometry that was used to define the subset. If you don’t want to plot the dataset, uncheck “Show plot of subset”.

  4. Change the name of the output file and specify a filepath if you wish. Then click “Create subset”. A shapefile will be saved for the subsetted dataset, and a plot will appear unless you turned the plotting option off.

Movie Maker#

movie_maker

App features#

With the Movie Maker App, you can

  • Create an animation file out of a stack of static maps that become frames in the animation.

  • Animations can include tracking data along with other layers, including environmental data from NASA Appeears, elevation contours, and other features. Frames using these layers can be created with ECODATA-Animate.

Using the app#
  1. Paste the full path to a folder of images to use as frames of the animation. You can create these frames using ECODATA-Animate. Make sure there are no other files in the folder.

  2. Select the frame rate for the video, in frames per second.

  3. Optionally, change the filename and filepath defining where to save the output.

  4. Click “Make movie”. The movie will be saved as a .mp4 file in the selected folder. Be sure not to change the folder location or name while the app is processing.

Tip

If you have issues with the video not playing properly, check whether it will play properly using VLC Media Player. It may be a format compatibility issue.

Environmental data#

Requesting environmental data from NASA#

You can use a .geojson file from the Tracks Explorer app to submit a request for environmental data from NASA’s Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) service.

  1. Go to NASA AppEEARS: https://appeears.earthdatacloud.nasa.gov/

  2. These data are free, but you will need an EarthData account to request data. Register at https://urs.earthdata.nasa.gov/users/new, and then sign in with your EarthData account.

  3. In the top menu bar, click “Extract > Area”.

  4. Click “Start new request”. The interface looks like this:

appeears_area_sample

  1. In the Extract Area Sample page, first upload the .geojson file you created in the Tracks Explorer. Alternatively, you can draw a polygon on the page or upload another .geojson or .shp file. It will display a preview of the area for which data will be provided.

  2. Select a date range. You can choose calendar dates for the start and end of the period, or select “Is Date Recurring?” to select a specific range of dates across years.

  3. Select the data layers you want. Type a product name (e.g., “MYD10A2”) or description (e.g., “snow”) and it will display a list summarizing relevant products. To review all options, available products are listed at https://appeears.earthdatacloud.nasa.gov/products. Also see the searchable catalog of many of these products at https://lpdaac.usgs.gov/product_search/?status=Operational. Once you have selected a product(s) from the list, click the “+” button on each desired layer. The layers will appear in the “Selected Layers” box to the right.

  4. In “Output options” at the bottom of the page, select “NetCDF-4” as the file format and “Geographic” as the projection, like this:

appeears_output_options

  1. Select “Submit”. Your submission will be added to a queue and you will receive emails from appeears-noreply@nasa.gov confirming the submission and notifying you when the data are ready for download. Keep in mind that notifications could end up in your spam box. The time needed to complete the request likely depends on the volume of data requested and the number of other requests their service has recently received. In our tests so far they have come through within a few hours.

  2. When your data are ready for download, you will receive an email from appeears-noreply@nasa.gov. Click the “Download” link in the email.

  3. The AppEEARS website will open in a new tab (log into your account if prompted) and you will see a list of files available to download. Supporting files are listed first, however you will need to download the resulting NetCDF (.nc) file for the next step in the ECODATA-Prepare app (Gridded Data Explorer).

  4. Check the box to the left of the file name and select Download > Download Files at the right. The downloaded file will appear in your Downloads folder, which you can then move to a local folder. We also recommend downloading at least the README.md file under Supporting files. This will document details of your request and results that you may need to return to later, including data quality information and how to cite your results.

appeears_download

Requesting environmental data from ECMWF#

ECMWF ERA5 hourly data can be requested here. Note that you will need to make an account to request data.

To request data:

  1. Go to the data request page.

  2. In the “Variable” section, select the variables you wish to download, for example as shown here:

ecmwf_variables

  1. Select the times of interest.

  2. Under “Geographical area”, select “Sub-region extraction” and enter the bounding box coordinates for the area of interest (max and min longitude and latitude).

  3. Under “Format”, select “NetCDF (experimental)”.

  4. If you have entered all required information and are logged in, you will have the option to “Submit Form” and send the request. An example of selections and options for steps 4-6 is shown here:

ecmwf_area_format

Support#

ECODATA is in the early stages of development, and any feedback is welcome! If you have any suggestions or feature requests, enounter any bugs, or come across places where the documentation is unclear, please submit a Gitub issue.

Developer guide#

Contributing#

Check out this simple guide for using git.

  1. Create a new branch for your contributions.

  2. Commit your changes in this branch.

  3. Open a pull request to merge changes from your branch into the repository’s develop branch.

Documentation#

Documentation for this project is created using Sphinx and is hosted at Read the Docs (https://ecodata-apps.readthedocs.io/). The source files for these pages are located in the docs/apps folder of the repository. To edit the documentation, edit the markdown files in this folder (or sub-folders). Note that the docs/index.md file specifies the contents for the docs site. If a sub-folder has a index.md file, that file specifies the contents for that section of the docs site (e.g. docs/user_guide/index.md). If files are added or removed, the corresponsing index files will also need to be updated.

Building the docs#

After editing the pages, you can look at a build of the pages to see how things will actually look in the docs website. There are two options for this:

  • Option 1: Open a pull request, and Read the Docs will build a preview of the docs pages. A link to the build can be found near the bottom of the page of the PR, in the merge checks section (once the build is finished, click on “Details” for the docs/readthedocs.org:ecodata-animate item. You may have to click “Show details” next to where it says “All checks have passed”). You can push additional commits to the open PR if you want to change anything after seeing the preview build.

  • Option 2: Build the docs locally. You will need to have python and the docs requirements installed.

    • To install the doc requirements: Developing Installation instructions

    • Then Run PROJECT=apps sphinx-build -b html docs docs/_build/apps from the repo’s root directory

    • To view the build, open the index.html in the docs/_build/apps directory that was created.

Versions of the docs#

  • Read the Docs builds multiple versions of the documentation (for different branches of the repository). In the bottom corner of the docs pages, there is a box indicating which version you are viewing. You can click on that box to pick a different version.

Developer installation#

  1. Clone the repository, and navigate to the root directory of the cloned repo

  2. Create the environment for the package:

    mamba env create --file ecodata-env.yml --name eco-dev
    
  3. Activate the eco-dev environment (this will install the package in editable mode):

    mamba activate eco-dev
    
  4. With the eco-dev environment activated, install the additional dev requirements (needed for docs, testing, code style, etc):

    mamba env update --name eco-dev --file ecodata-dev-env.yml
    

Indices and tables#