Welcome to ECODATA-Prepare!#

Overview#

ECODATA-Prepare is a set of Python-based apps to access and process remote sensing and other environmental data products and prepare them for further use along with animal movement data. ECODATA-Prepare is designed to be along with the ECODATA-Animate tool to create movies of animal movement data, and can also be used to create input to resource, habitat and step selection models. Development is supported by MathWorks® and the NASA Earth Science Division, Ecological Forecasting Program, as part of Room to Roam: Y2Y Wildlife Movements project.

Within ECODATA-Prepare,

  • The Tracks Explorer App visualizes movement track location points and point densities with several static maps, allows selecting a spatial frame around the track points that is relevant for analysis, and prepares a .geojson file that can be used in the NASA EARTHDATA AppEEARS interface to extract many different remote sensing data products within this research area. This app also provides the longitude-latitude coordinates of the frame around the research area, which can be used to request data from ECMWF or many other environmental data sources, which can be converted to NetCDF format if needed for further steps described below.

  • The Gridded Data Explorer App allows you to interpolate and subset the temporal and spatial resolutions of environmental data in the form of a temporal stack (or single static map) NetCDF file. It can also read in several other data formats (which could be resaved as NetCDF). You can also read GIS polygons as shapefiles and mask the environmental data outside or inside the polygons. It subsequently calculates data summaries by period and polygon.

  • The Subsetter App allows you to clip relevant features out of large GIS files.

  • The Movie Maker App produces an animation file out of a stack of static maps which were produced by the ECODATA-Animate program.

We recommend the following workflow (you can skip any steps you don’t need):

  1. Use the Tracks Explorer App to create a .geojson file that defines the area of interest based on your tracking data.

  2. Use this file to request environmental data from NASA AppEEARS to use in subsequent animations or analysis.

  3. Use the Gridded Data Explorer App to review and further process the environmental data received from NASA AppEEARS.

  4. Use the Subsetter App to extract relevant features from very large datasets (for example, global vector GIS data products for features like roads, water bodies or protected areas) to use in subsequent animations or analysis.

  5. Use ECODATA-Animate to combine your tracking data, environmental data, subsetted vector layers, and other information, configure viewing options, and create a set of images to use as frames for an animation or for further exploration and analysis.

  6. Use the Movie Maker App to create an animation from the output of ECODATA-Animate.

For help, submit a GitHub issue or contact support@movebank.org.

Python package#

The functions underlying the ECODATA-Prepare apps can also be used directly as a python package (i.e. without the GUI inferface). The package documentation is here.

Contents#

Installing the apps#

Installers#

Download installers for the latest release here, available for Mac, Windows, and Linux. You can also see all releases (including previous versions and pre-releases) here.

  1. Download the installer for your operating system, and extract the files from the zipped folder.

  2. Run the installer and follow the prompts. You can change the default installation location if you wish. This installation location is where the program files will be installed (i.e., files not intended to be accessed by the user after installation). If you have a previous version already installed in the default location, you will need to install the new version in a different location, or remove the previous version before installing the new version.

  3. A file used to launch the apps (ecodata.sh for Mac and ecodata.bat for Windows) will be copied to your Downloads folder. Double-click this file to launch the apps.

  4. A terminal will open indicating that the apps are launching. A window will open on your default web browser, showing the main app gallery page (the apps are running locally at localhost:5006). There may be a short wait the first time you launch the apps, or the first time you launch after an update.

  5. You may receive a message “Do you want the application “python3.9” to accept incoming network connections?” You can click Allow.

  6. Keep the terminal application you used to launch the program open while running the app. To shut down the apps, close the terminal that you used for launching.

Alternative installation methods#

The apps can also be installed from the command line. If you have issues with the first installation method (e.g. your computer’s security settings are preventing the installer from running), you can use this method as a backup.

Launching ECODATA-Prepare from the terminal#

Launch the ECODATA-Prepare apps using the command below. For Mac/Linux, you will run the command below from the built-in Mac “Terminal” application, or any other terminal app. For Windows, you will run the command using the Miniforge Prompt. If this is not already open, find it by searching for “Miniforge Prompt” in the start menu.

mamba activate eco
python -m ecodata.app

A window will open on your default web browser, showing the main app gallery page (the apps are running locally at localhost:5006). There may be a short wait the first time you launch the apps, or the first time you launch after an update.

You may receive a message “Do you want the application “python3.9” to accept incoming network connections?” You can click Allow.

Keep the terminal application you used to launch the program open while running the app. To shut down the apps, close the terminal that you used for launching.

Updating ECODATA-Prepare from the terminal#

You can check which version of ecodata you have installed by running:

mamba activate eco
mamba list ecodata

You can see new releases on the releases page

If you want to install a new release, you can update your version of the apps by running the command below:

mamba activate eco
mamba update ecodata

Install using Anaconda (alternative method)#

See below for steps to install and run the ECODATA-Prepare apps. For Mac/Linux, you will run the commands below from the built-in Mac “Terminal” application, or any other terminal app. For Windows, you will install Anaconda and then run the commands below using the Anaconda Powershell Prompt. You can launch this by searching for “Anaconda Powershell” in the start menu, or by launching Anaconda Navigator and looking for “Powershell Prompt” in the home screen.

Note

The apps are still under development, and installation will be simplified soon!

Installing Anaconda#

  1. Anaconda Distribution is an free, open-source distribution platform for managing and deploying Python and R packages. Install Anaconda if you do not have it on your computer. Download the installer here and then run the file to install the program.

  2. For Windows only: To address compatibility issues, downgrade conda to the previous version (v4.14). In the Anaconda Powershell prompt, run:

    conda install conda=4.14

Preparing the Python environment and installing ECODATA-Prepare#

  1. Download the conda environment file here. This file specifies the Python package requirements used by the apps, and can be deleted following installation.

  2. Open a terminal (for Windows, use Anaconda Powershell).

  3. Navigate to the directory where you saved the environment file. Run cd <path_to_directory> (note that you need to replace the path with the actual path to the directory where you saved the file), for example:

    cd Downloads

  4. You can confirm you navigated to the correct directory by running “ls” to list the files in that directory:

    ls

    You should see the environment file in the output, for example:

        Directory: C:\Users\jmissik\Downloads


Mode                LastWriteTime       Length Name
----                -------------       ------ ----
-a----      11/30/2022  12:38 PM        23470 environment-clean-build.yml

If your directory has many files, replace “ls” with “ls -lt” to sort files by the time they were last modified.

  1. Install the environment (note that if your environment file is not called “environment-clean-build.yml”, you need to replace this name with the actual name of your file):

    conda env create --file environment-clean-build.yml --name eco

    This will download all of the packages for the application, so it may take some time to finish, depending on your internet speed. Once the environment is successfully created, the terminal output should look something like:

    done
#
# To activate this environment, use
#
#   $ conda activate eco
#
# To deactivate an active environment, use
#
#   $ conda deactivate

    If there were issues with the installation, you can try using the locked environment file for your operating system instead: Windows | Intel Mac | M1 Mac

  2. After the Python environment is completed, you need to install the ECODATA-Prepare application itself. From the terminal, run:

    conda activate eco
pip install git+https://github.com/jemissik/ecodata.git@main

User guide#

ECODATA-Prepare is a set of four Python-based apps to read, process, and create animations from animal tracking data and gridded environmental data. Read here for an overview and to learn how to install, contribute and get support.

Want a downloadable version of the docs? This documentation is also available for download as a PDF, Epub, or zipped HTML.

Getting started#

Before using this program, download a local copy of the tracking data you want to use, in Movebank’s .csv format. It is possible to subset or combine data from different studies. Read the installation instructions to install or update the program.

Follow the instructions below to launch the program. A window will open on your default web browser, showing the app gallery. The apps are running locally at localhost:5006. Note: Anaconda gives no indication the command is running. There may be a short wait (10+ seconds) before the window opens.

When it opens, it will display the main panel, showing the four apps. Click on an app to launch it. From there, you can navigate between apps within the interface, or switch between them by pasting these URLs in your browser window:

Launch in Windows#

  • Open Anaconda PowerShell.

  • Copy-paste the below code into the prompt window, and press Enter:

conda activate eco
python -m ecodata.app

Launch on Mac#

  • Open Anaconda Navigator.

  • Navigate to “eco”.

anaconda_navigator

  • Hit the play button and select “Open Terminal”.

  • A Terminal window will open. Enter the following and hit Return:

python -m ecodata.app

ecodata-prepare_panel

Apps#

Tracks Explorer#

tracks_explorer_map

App features#

With the Tracks Explorer App, you can

  • Visualize point locations from your Movebank data as individual points or aggregated point density. You can select from a variety of map tiles to use as a background.

  • Automatically generate a spatial frame around the track points and save this as a .geojson file that can be used in the NASA AppEEARS interface to extract remote sensing products for the research area.

Using the app#

tracks_explorer_start Data from Hebblewhite and Merrill, study Hebblewhite Alberta-BC Wolves

  1. If you haven’t already, download a local copy of the tracking data you want to use, in Movebank’s .csv format. It is possible to subset or combine data from different studies.

  2. Launch the Tracks Explorer App.

  3. Under “Select File”, paste the full filepath to a .csv file of tracking data, including the file name. In the current version, this .csv file needs to use Movebank’s format. An example filepath is “/Users/myname/MovebankData/wolves.csv”.

  4. Click “Load data”. An interactive map will appear of your dataset, where you can pan/zoom to explore the dataset. If a map does not load, view the status message. The message “Error.. Check options and try again” likely indicates that your filepath is incorrect.

  5. You can change the background map tile using the drop-down menu in the sidebar.

  6. Toggle the “Datashade tracks” checkbox in the sidebar to turn on/off the point density aggregation feature. When unchecked, individual semi-transparent points will be displayed. Note that for very large datasets, plotting will be slow if the “Datashade tracks” option is turned off.

  7. A boundary for the tracks dataset is shown on the map in red. The default option uses a rectangular boundary with a buffer size of 0.1 (scale relative to the extent of the tracks). You can adjust the buffer size or change the boundary shape to a convex hull using the widgets in the sidebar.

  8. To save the boundary to a .geojson file, scroll down and if needed, edit the file name and location under “Output file”. Then simply click the “Save extent” button.

Requesting environmental data from NASA#

You can use a .geojson file from the Tracks Explorer app to submit a request for environmental data from NASA’s Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) service.

  1. Go to NASA AppEEARS: https://appeears.earthdatacloud.nasa.gov/

  2. These data are free, but you will need an EarthData account to request data. Register at https://urs.earthdata.nasa.gov/users/new, and then sign in with your EarthData account.

  3. In the top menu bar, click “Extract > Area”.

  4. Click “Start new request”. The interface looks like this:

appeears_area_sample

  1. In the Extract Area Sample page, first upload the .geojson file you created in the Tracks Explorer. Alternatively, you can draw a polygon on the page or upload another .geojson or .shp file. It will display a preview of the area for which data will be provided.

  2. Select a date range. You can choose calendar dates for the start and end of the period, or select “Is Date Recurring?” to select a specific range of dates across years.

  3. Select the data layers you want. Type a product name (e.g., “MYD10A2”) or description (e.g., “snow”) and it will display a list summarizing relevant products. To review all options, available products are listed at https://appeears.earthdatacloud.nasa.gov/products. Also see the searchable catalog of many of these products at https://lpdaac.usgs.gov/product_search/?status=Operational. Once you have selected a product(s) from the list, click the “+” button on each desired layer. The layers will appear in the “Selected Layers” box to the right.

  4. In “Output options” at the bottom of the page, select “NetCDF-4” as the file format and “Geographic” as the projection.

  5. Select “Submit”. Your submission will be added to a queue and you will receive emails from appeears-noreply@nasa.gov confirming the submission and notifying you when the data are ready for download. The time needed to complete the request likely depends on the volume of data requested and the number of other requests their service has recently received. In our tests so far they have come through within a few hours.

  6. When your data are ready for download, click the “Download” link in the email you received from appeears-noreply@nasa.gov.

  7. The Appeears website will open in a new tab (log into your account if prompted) and you will see a list of files available to download. Supporting files are listed first, however you will need to download the resulting NetCDF (.nc) file for the next step in the ECODATA-Prepare app (Gridded Data Explorer).

  8. Check the box to the left of the file name and select Download > Download Files at the right. The downloaded file will appear in your Downloads folder, which you can then move to a local folder. We also recommend downloading at least the README.md file under Supporting files. This will document details of your request and results that you may need to return to later, including data quality information and how to cite your results.

appeears_download

Gridded Data Explorer#

gridded_data_explorer

App features#

With the Gridded Data Explorer App, you can

  • Read, interpolate and subset the temporal and spatial resolutions of environmental data in the form of a temporal stack (or single static map) NetCDF file.

  • Read GIS polygons and mask the environmental data outside or inside of the polygons.

  • Calculate data summaries by time period and spatial area.

  • Save processed environmental data and summary statistics for further use.

Using the app#

Tip

Throughout the App, scroll down to view status messages indicating application progress and next steps.

  1. Under “Input environmental dataset”, paste the full filepath to a NetCDF (.nc) file received from NASA AppEEARS (see instructions) or any other source. The filepath should include the file name. An example filepath is “/Users/myname/RS_Data/NDVI.nc”.

  2. Click “Load data”. If it is able to read the file, the app will load available variables from the file into dropdown lists for Latitude, Longitude, Time and Variable of interest, and will attempt to select the correct variables automatically. If nothing happens, you might need to correct the filepath.

  3. If the app does not assign the correct variable names from the file, you can specify them manually by clicking on the field and selecting from the dropdown list.

  4. In the “Variable of interest” dropdown, select the variable you would like to explore.

gridded_data_explorer_input

  1. Once the input parameters are set, click “Update variable names”. Plots of the data will appear.

  2. At the top of the area with the plots, there are two tabs (“Charts” and “Data”). If you click the “Data” tab, you can see more information about the file contents, including the dimensions of the dataset and the variables in the file. Click on the “Show/Hide attributes” icon to the right of a variable to view variable metadata including long names, units, and legend classes.

gridded_data_explorer_data

  1. In the “Charts” tab, there are a variety of options to explore the selected variable in space and time.

  2. Filter by area: Under “Input polygon file”, you can input a polygon file to select or mask data inside polygon boundaries. Selecting will keep data inside polygons, and masking will keep data outside polygons. Paste the full filepath to a polygon file (e.g., .shp), including the file name, and click “Load file”. When the file loads, it will be plotted on the Chart view above. Select and option and then click “Update filters”. To remove the filter, click “Revert filters”.

  3. Filter by time: To filter the data for specific times, use the options under “date range selection” under Selection Options on the sidebar. There are time filtering options in the sidebar if you want to filter the data for a specific time range or time conditions (e.g., daytime only, specific seasons or months). You can use the time slider bars to filter by a range of values or enter discrete values into the box. For example, to filter by month you would click the “Month” tab above the “Range Values” option and either use the sliding bar to restrict your data to months of interest or enter numeric months into the Discrete Values box (note that when you click in this box, a dropdown appears with the month values from your Appeears dataset). Available ranges and categories for discrete filtering will depend on the content of your dataset. Select your options and then click “Update filters”. If you want to remove the filters you have applied, click “Revert filters”.

  4. Resample by time: It is possible to resample the data by time. The app will estimate attribute values from the original values at the requested time schedule by interpolating when upscaling, or using means for downsampling. Upscaling could be used to show smooth transitions in an animation with daily timestamps of a product with a 16-day resolution; downscaling could be used to reduce storage and processing needs for a large hourly dataset if you need only daily data. Under “Processing Options” in the sidebar, select the desired time frequency and click “Resample time”. Once processing is complete, the status at the bottom of the page will show “Dataset Resampled”.

  5. Coarsen spatial resolution: It is possible to coarsen (downsample) the spatial resolution of the data using block aggregation, for example to reduce file sizes or processing time for subsequent uses. Under “Processing options” in the sidebar, select a window size for coarsening and click “Coarsen dataset”. The window size will indicate the number of pixels in the x and y directions over which a mean should be calculated. Once processing is complete, the status at the bottom of the page will show “Aggregation completed”.

  6. After applying filters or processing to the data, you can save the results as a new .nc file. Under “Output file”, choose a name for the file. You can specify a filepath, or provide only a file name to have it saved in a default folder on your computer. Select “Save dataset”. Below, a message “File saved to: [filepath]” will indicate where the file is stored.

  7. Calculate summary statistics: First select grouping options in the “Calculate statistics” panel, then click “Calculate statistics”.

gridded_data_explorer_data_stats_filters

Once processing is complete, the status at the bottom of the page will show “Calculations completed”, and a table of the summary statistics will appear in the “Statistics” panel. You can save this output as a new .csv file. Under “Output file for statistics”, choose a name for the file. You can specify a filepath, or provide only a file name to have it saved in a default folder on your computer. Select “Save statistics”. The status message will update to indicate where the file is stored.

gridded_data_explorer_data_stats_results

Subsetter#

subsetter

App features#

With the Subsetter App, you can

  • Clip relevant features out of large shapefiles based on a bounding box, the extent of a tracking dataset, or a custom polygon.

  • Save the extracted subset for further use.

Using the app#

  1. In the “GIS file” field, paste the full filepath to a GIS dataset (e.g., a shapefile or file geodatabase) from which you want to extract a subset. The filepath should include the file name, or if the dataset consists of multiple files (e.g., for a shapefile) the folder in which the files are saved. An example filepath is “/Users/myname/global_roads_dataset.gdb”.

  2. Next, define the spatial area for which you want to extract data from the GIS dataset. You can do this using a bounding box, track points, or a bounding geometry. There are specific settings for each option:

    • Bounding Box: Specify the latitude and longitude range for a rectangular bounding box. Provide coordinates in decimal degrees, WGS84 coordinate reference system.

    • Track Points: Provide the full filepath, including the filename, for a .csv file of tracking data containing location-long and location-lat in decimal degrees, WGS84 coordinate reference system. The app will draw a boundary around the track data: choose whether to do this using a rectangular or convex hull method, and optionally define a buffer around which to expand the boundary. This buffer is relative to the extent of the tracking data (the selected buffer size times the greater of the north-south or east-west extent).

    • Bounding Geometry: Provide the full filepath, including the filename, for a file (.geojson, .shp or .gdb) defining a polygon. This could include output of the Tracks Explorer App.

  3. For any of the options above, decide whether to clip features that overlap with the edge of the spatial area defined in the previous step. Uncheck “Clip features at boundary edge” to retain complete lines or polygons that overlap with the boundary edge. By default, the app will plot the data subset once it is created, along with the geometry that was used to create the subsetting boundary, and the actual geometry that was used to define the subset. If you don’t want to plot the dataset, uncheck “Show plot of subset”.

  4. Change the name of the output file and specify a filepath if you wish. Then click “Create subset”. A shapefile will be saved for the subsetted dataset, and a plot will appear unless you turned the plotting option off.

Movie Maker#

movie_maker

App features#

With the Movie Maker App, you can

  • Create an animation file out of a stack of static maps that become frames in the animation.

  • Animations can include tracking data along with other layers, including environmental data from NASA Appeears, elevation contours, and other features. Frames using these layers can be created with ECODATA-Animate.

Using the app#

  1. Paste the full path to a folder of images to use as frames of the animation. You can create these frames using ECODATA-Animate. Make sure there are no other files in the folder.

  2. Select the frame rate for the video, in frames per second.

  3. Optionally, change the filename and filepath defining where to save the output.

  4. Click “Make movie”. The movie will be saved as a .mp4 file in the selected folder. Be sure not to change the folder location or name while the app is processing.

Tip

If you have issues with the video not playing properly, check whether it will play properly using VLC Media Player. It may be a format compatibility issue.

Support#

ECODATA is in the early stages of development, and any feedback is welcome! If you have any suggestions or feature requests, enounter any bugs, or come across places where the documentation is unclear, please submit a Gitub issue.

Developer guide#

Contributing#

Check out this simple guide for using git.

  1. Create a new branch for your contributions.

  2. Commit your changes in this branch.

  3. Open a pull request to merge changes from your branch into the repository’s develop branch.

Documentation#

Documentation for this project is created using Sphinx and is hosted at Read the Docs (https://ecodata-apps.readthedocs.io/). The source files for these pages are located in the docs/apps folder of the repository. To edit the documentation, edit the markdown files in this folder (or sub-folders). Note that the docs/index.md file specifies the contents for the docs site. If a sub-folder has a index.md file, that file specifies the contents for that section of the docs site (e.g. docs/user_guide/index.md). If files are added or removed, the corresponsing index files will also need to be updated.

Building the docs#

After editing the pages, you can look at a build of the pages to see how things will actually look in the docs website. There are two options for this:

  • Option 1: Open a pull request, and Read the Docs will build a preview of the docs pages. A link to the build can be found near the bottom of the page of the PR, in the merge checks section (once the build is finished, click on “Details” for the docs/readthedocs.org:ecodata-animate item. You may have to click “Show details” next to where it says “All checks have passed”). You can push additional commits to the open PR if you want to change anything after seeing the preview build.

  • Option 2: Build the docs locally. You will need to have python and the docs requirements installed.

    • To install the doc requirements: use the ecodata-dev-env.yml file in the root of the repository.

    • Build the docs: Run PROJECT=apps sphinx-build -b html docs docs/_build/apps from the repo’s root directory

    • To view the build, open the index.html in the docs/_build/apps directory that was created.

Versions of the docs#

  • Read the Docs builds multiple versions of the documentation (for different branches of the repository). In the bottom corner of the docs pages, there is a box indicating which version you are viewing. You can click on that box to pick a different version.

Developer installation#

  1. Clone the repository, and navigate to the root directory of the cloned repo

  2. Create the conda environment for the package:

    conda env create --file ecodata-env.yml --name eco-dev

  3. Activate the eco-dev environment (this will install the package in editable mode):

    conda activate eco-dev

  4. With the eco-dev environment activated, install the additional dev requirements (needed for docs, testing, code style, etc):

    conda env update --name eco-dev --file ecodata-dev-env.yml --prune

Indices and tables#