Downloading HYCOM data using Matlab and OPeNDAP

I am trying to download HYCOM (www.hycom.org) data using the OPeNDAP access method and MATLAB. The data url can be generated by selecting desired variables from this link. I need to download the data using scripts to automate the download of many years of data.

My MATLAB code looks like below, which you should be able to run as well without any modification:

% This script download the Global HYCOM reanalysis data for a given time step
for i=0:2863
timestep = 1; % This value need to be adjusted for duration of the record needed
j=i+timestep;
url = ['http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/1994?depth[0:1:39],lat[0:1:3250],lon[0:1:4499],time['...
    num2str(i) ':1:' num2str(j) ...
    '],tau[0:1:2864],water_u[0:1:0][0:1:0][0:1:0][0:1:0],water_u_bottom[0:1:0][0:1:0][0:1:0],water_v[0:1:0][0:1:0][0:1:0][0:1:0],water_v_bottom[0:1:0][0:1:0][0:1:0],water_temp[0:1:0][0:1:0][0:1:0][0:1:0],water_temp_bottom[0:1:0][0:1:0][0:1:0],salinity[0:1:0][0:1:0][0:1:0][0:1:0],salinity_bottom[0:1:0][0:1:0][0:1:0],surf_el[0:1:0][0:1:0][0:1:0]'];
 
info = ncinfo(url);
    
% Successfully read these variables 
depth = ncread(url, 'depth');  
lat = ncread(url, 'lat');
lon = ncread(url, 'lon');
time = ncread(url, 'time');
    
% Trouble reading below variables
water_u = ncread(url, 'water_u');
water_v = ncread(url, 'water_v');
water_u_bottom = ncread(url, 'water_u_bottom');
water_v_bottom = ncread(url, 'water_v_bottom');
surf_el = ncread(url, 'surf_el');
water_temp = ncread(url, 'water_temp');

end

The scripts reads the depth, lat, lon, time fine, but gives error in reading the water_u variable on wards. The error message is:

Error using netcdflib
The NetCDF library encountered an error during execution of 'getVarShort' function - 'Index exceeds dimension bound
(NC_EINVALCOORDS)'.
Error in netcdf.getVar (line 136)
data = netcdflib(funcstr,ncid,varid,varargin{:});
Error in internal.matlab.imagesci.nc/read (line 605)
data = netcdf.getVar(gid, varid);
Error in ncread (line 58)
vardata = ncObj.read(varName, varargin{:});

I am not sure what is wrong with my code, and couldn't find any answer searching online so far. Appreciate your help.

[Belinda Finlay]

Did you solve this as I am having the same issue (although I got the surf_el values)?

Answer 1

[Amitava Guha]

I found out that downloading data from HYCOM is easier using Powershell script. Below is my code for a single grid point and for one year. I have been successful in running 5 such scripts at the same time, downloading 5 years of data for 1 grid point in 4-6 hour timeframe.

# This script is used to download current hindcast data from www.hycom.org 

# Result directory
# --------------------------------------------------------
$resultDirectory = $PSScriptRoot + "\Data"
If (!(test-path $resultDirectory))
{
    md $resultDirectory
}

# Input
# -------------------------------------------------------
# Select coordinate for region of interest
# Location: Latitude 6° 20' S , Longitude: 11° 15' E

#East
$east = 11.30

#West
$west = 11.25

#South
$south = -6.33

#North
$north = -6.31

$date_start = '01-Jan-1994 12:00:00'
$date_end = '31-Dec-1994 23:00:00' 

# --------------------------------------------------------
# Converting dates to datetime
$startDate = [datetime]::ParseExact($date_start,'dd-MMM-yyyy HH:mm:ss',$null)
$endDate = [datetime]::ParseExact($date_end,'dd-MMM-yyyy HH:mm:ss',$null)

Write-Host "Downloading data from " $startDate.ToString('yyyy-MM-ddTHH:mm:ssZ') " to "  $endDate.ToString('yyyy-MM-ddTHH:mm:ssZ')


for ($time = $startDate; $time -le $endDate; $time=$time.AddHours(3)){

$error_flag = 1
while ($error_flag -eq 1){
	
	# Example url. Do not delete. Used for date time format reference purposes
	#$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/2015?var=water_u&var=water_v&north=8.6&west=-57.3&east=-57.25&south=8.5&time=2015-01-01T18:00:00Z&accept=netcdf4" 

	# Download url
	$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/" + $time.ToString('yyyy') + "?var=water_u&var=water_v&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# Output file name
	$fileName = $time.ToString('yyyyMMdd_HH') + ".nc"
	$output = $PSScriptRoot + "\Data\" + $fileName


	Try{
		# Creating a web client which has the download file functionality
		#WebProxy = New-Object System.Net.WebProxy("hoeprx01.na.xom.com:8080",$true)
		$WebClient = New-Object System.Net.WebClient
		#$WebClient.Proxy=$WebProxy
		$WebClient.DownloadFile($url,$output)
		Write-Host "Successfully downloaded file:"  $fileName
		
		$error_flag = 0
	}
	Catch {
		Write-Host $_.Exception.Message`n
		$error_flag = 1

		Write-Host  "Retrying downloading file:" $fileName " ...."
	}

}

}

[Muhammad Nadzrin Nazri]

Dear Amitava,

Can you guide me to download using power shell? My area (south china sea):

North: 14

South: 0

East: 99

West: 117

I want 1 hourly, surface data from jan 2019 - Aug 2020:

//ncss.hycom.org/thredds/ncss/grid/GLBy0.08/expt_93.0/sur

[Gregory Pelletier]

Dear Amitava,

This powershell script is fantastic! It is the best script I have found yet for extracting data from hycom 53.X

I have a question about sometimes when it crashes if the file on the hycom server is missing. Some times when the file is missing for a particular datetime stamp the script gets stuck retrying and gives the error "The underlying connection was closed: An unexpected error occurred on a receive." When that happens I need to restart my laptop and try again from the next non-missing output datetime stamp.

Have you encountered that problem, and is there a new version of the script with some code to get around the problem of the missing datetime stamps on the hycom server?

Thanks for sharing this great script!

Best

Greg Pelletier

[Gregory Pelletier]

Instead of restarting the laptop, you can re-initialize the powershell session with the following two commands before submitting the rest of the script after an error loop occurs:

Clear-Host

Get-PSSession | Remove-PSSession

[Amitava Guha]

Dear Greg,

Glad you found my script useful. I did encounter issues you mentioned above when there is no data available for a given timestamp and had to restart the code. For latest years, you can change the $url to anothe experiment number as shown below:

$url = "https://ncss.hycom.org/thredds/ncss/GLBy0.08/expt_93.0/uv3z?var=water_u&var=water_v&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

Best,

Amitava

[Gregory Pelletier]

Dear Amitava,

Thank you very much. This worked perfectly for 2016-present.

I made the following modified version of the Powershell script that is more generalized and has all of the $url selections to use for any year between 1994 to the present. This modified version also lets the user select any of the hycom variables to download. I also added a short section with instructions to aid the user

Best

Greg

# ----------
# get_hycom_3hr.ps1 
# 
# Microsoft Powershell script to extract 3-hr hycom model output
# This script is used to download hycom hindcast data from www.hycom.org 

# Original script by Amitava Guha
# Modified by Greg Pelletier ([e-mail]) 
# ----------

# - - -
# INSTRUCTIONS
#
# 1) In the user input section above the while loop, specify the following:
# 		- specify the list of variables to be extracted from any combination of $var_list = "surf_el,water_temp,salinity,water_u,water_v"
# 		- specify the $west, $east, $south, and $north extent of the bounding box to be extracted
#  		- specify the name of the $resultDirectory where the hycom data will be saved as output
# 		- specify the $date_start and $date_end of the period to be extracted
# 2) In the user selection section within the while loop, select the appropriate $url for the time period and variables to be extracted
# 3) In the Windows search box, search for "Powershell" and open a Powershell window, 
#    and then copy and paste the entire contents of this script into the Powershell window to execute.
# 	 If you do not have Windows, you may use this script in the new cross-platform version of PowerShell 
# 	 that is available for MacOS and Linux at the following link from Microsoft: https://aka.ms/pscore6
# 4) During execution you sould see the progress of each 3-hour file that is extracted during the period of interest 
#    from beginning to end. Each nc file name has the format yyyyMMdd_HH.nc to indicate the datetime stamp in UTC
#
# Occasionaly there may be times when the hycom server is not responsive 
# and the script may try several times to download a particular 3-hour time.
# Each 3-hour time is saved in a separate netcdf file in the specified output folder. 
# Sometimes if the hycom data are missing there may appear to be an endless loop of trials to download the missing data.
# In those cases it may be necessary to stop the script and change the $date_start to a value that resumes with non-missing data
# and then re-run the script with that new start time.
#
# Sometimes problems with endless error loops may be fixed by re-initialize a new powershell session with the following
# two commands at the Powershell prompt before submitting the rest of the script in the powershell window:
Clear-Host 							
Get-PSSession | Remove-PSSession	

# - - -
# VERSION HISTORY
#
# v06 changed the default bounding box to Parker's LiveOcean open boundary box
# v05 added option for $var_list in user inputs
# v04 $url choices for all of the experiments in the same script to allow use for any time period from 1994-present

# - - -
# This main functional parts of this script were obtained from the following link: 
# https://www.mathworks.com/matlabcentral/answers/542831-downloading-hycom-data-using-matlab-and-opendap
# by Amitava Guha on 04-Jan-2021
# modified by Greg Pelletier 10-Jan-2023

# - - -


# ----------

# ----------
# ----------
# USER INPUT SECTION

# NOTE: In addition to the user inputs in this section, the user also must select the appropriate url in the while loop 
# by commenting/uncommenting the appropriate $url lines of code as needed from the choices that are provided below

# - - -
# uncomment one of the following choices for $var_list, or input any other subset of variables:
$var_list = "surf_el,water_temp,salinity,water_u,water_v"
# $var_list = "water_temp,salinity,water_u,water_v"
# $var_list = "water_temp,salinity"
# $var_list = "water_u,water_v"

# - - -
# input the bounding box west, east, south, and north for region to extract
# to download single grid points, use $east-$west and $south-$north inputs that are tiny differences, like +/- .04 difference

# Defaults below are Parker MacCready's LiveOcean Model boundary hycom extraction box:
$west = 360 - 131			# 0 to 360 degE
$east = 360 - 121			# 0 to 360 degE
$south = 39					# -80 to 80 degN
$north = 53					# -80 to 80 degN

# - - -
# input the directory where the extracted nc files are saved
# $resultDirectory = $PSScriptRoot + "\data"
$resultDirectory = "c:\data\hycom\3hr\2022\"
# $resultDirectory = "/mnt/c/data/hycom/3hr/2022/"

# - 
# input the start and end datetime stamp (edit below as needed)
# Note: all years start on at 12:00 UTC on 01-Jan except as noted below regarding missing data

# -  
# 1994-2015 (edit below as needed to extract up to one year at a time between 1994-2015)
# Use GLBv0.08 expt 53.X for 1994-2015:
# $date_start = '01-Jan-2015 12:00:00'
# $date_end = '31-Dec-2015 21:00:00'

# -  
# 2016 (1/1/2016 to 4/30/2016)
# Use GLBv0.08/expt_56.3 for 1/1/2016 or 7/1/2014 to 4/30/2016, Base Time: 2014-07-01T12:00:00Z
# $date_start = '01-Jan-2016 12:00:00'
# $date_end = '30-Apr-2016 21:00:00'

# -  
# 2016 (5/1/2016 to 12/31/2016) and 2017 (1/1/2017 to 1/31/2017)
# Use GLBv0.08/expt_57.2 for 5/1/2016 to 1/31/2017, Base Time: 2016-05-01T12:00:00Z
# NOTE: manually move the Jan-2017 data to the 2017 output folder after downloading it to the 2016 folder
# $date_start = '01-May-2016 12:00:00'
# $date_end = '31-Jan-2017 21:00:00'

# -  
# 2017 (2/1/2017 to 5/31/2017)
# Use GLBv0.08/expt_92.8 for 2/1/2017 to 5/31/2017, Base Time: 2017-02-01T12:00:00Z
# $date_start = '01-Feb-2017 12:00:00'
# $date_end = '31-May-2017 21:00:00'

# -  
# 2017 (6/1/2017 to 9/30/2017)
# Use GLBv0.08/expt_57.7 for 6/1/2017 to 9/30/2017, Base Time: 2017-06-01T12:00:00Z
# $date_start = '01-Jun-2017 12:00:00'
# $date_end = '30-Sep-2017 21:00:00'

# -  
# 2017 (10/1/2017 to 12/31/2017)
# Use GLBv0.08/expt_92.9 for 10/1/2017 to 12/31/2017, Base Time: 2017-10-01T12:00:00Z
# $date_start = '01-Oct-2017 12:00:00'
# $date_end = '31-Dec-2017 21:00:00'

# -  
# 2018
# Use GLBv0.08/expt_93.0 for 1/1/2018 to 12/31/2018 or 2/18/2020, Base Time: 2018-01-01T12:00:00Z
# $date_start = '01-Jan-2018 12:00:00'
# $date_end = '31-Dec-2018 21:00:00'

# -  
# 2019-present (edit below as needed to extract up to one year at a time)
# Use GLBy0.08 expt 93.0 for 2019-present:

# 2019
# $date_start = '01-Jan-2019 12:00:00'
# $date_end = '31-Dec-2019 21:00:00'

# 2020
# $date_start = '01-Jan-2020 12:00:00'
# $date_end = '31-Dec-2020 21:00:00'

# 2021
# $date_start = '01-Jan-2021 12:00:00'
# $date_end = '31-Dec-2021 21:00:00'

# 2022
$date_start = '01-Jan-2022 12:00:00'
$date_end = '31-Dec-2022 21:00:00'


# END OF USER INPUTS
# ----------
# ----------

# ----------

# make a new directory to store the output results if it does not exist already
If (!(test-path $resultDirectory))
{
    md $resultDirectory
}

# - - -
# Converting dates to datetime
$startDate = [datetime]::ParseExact($date_start,'dd-MMM-yyyy HH:mm:ss',$null)
$endDate = [datetime]::ParseExact($date_end,'dd-MMM-yyyy HH:mm:ss',$null)
Write-Host "Downloading data from " $startDate.ToString('yyyy-MM-ddTHH:mm:ssZ') " to "  $endDate.ToString('yyyy-MM-ddTHH:mm:ssZ')

# - - -
# the following for loop does the extraction

for ($time = $startDate; $time -le $endDate; $time=$time.AddHours(3)){
$error_flag = 1
while ($error_flag -eq 1){

	# - - -

	# - - -
	# - - -
	# USER SELECTION OF $url (uncomment one of the $url choices below for the time period being downloaded)

	# - - -
	# # 1994-2015 (up to one year at a time)

	# # Use GLBv0.08 expt 53.X for 1994-2015
	# $url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/" + $time.ToString('yyyy') + "?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# - - -
	# # 2016-2018 (note that 2016 and 2017 are split into several experiments, the user must choose the appropriate experiment for the dates being extracted)

	# -  
	# # 2016 (1/1/2016 to 4/30/2016)
	# # Use GLBv0.08/expt_56.3 for 7/1/2014 to 4/30/2016, Base Time: 2014-07-01T12:00:00Z
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_56.3?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# -  
	# # 2016 (5/1/2016 to 12/31/2016) and 2017 (1/1/2017 to 1/31/2017)
	# # Use GLBv0.08/expt_57.2 for 5/1/2016 to 1/31/2017, Base Time: 2016-05-01T12:00:00Z
	# # NOTE: manually move the Jan-2017 data to the 2017 folder after downloading it to the 2016 folder
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_57.2?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# -  
	# # 2017 (2/1/2017 to 5/31/2017)
	# # Use GLBv0.08/expt_92.8 for 2/1/2017 to 5/31/2017, Base Time: 2017-02-01T12:00:00Z
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_92.8?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# -  
	# # 2017 (6/1/2017 to 9/30/2017)
	# # Use GLBv0.08/expt_57.7 for 6/1/2017 to 9/30/2017, Base Time: 2017-06-01T12:00:00Z
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_57.7?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# -  
	# # 2017 (10/1/2017 to 12/31/2017)
	# # Use GLBv0.08/expt_92.9 for 10/1/2017 to 12/31/2017, Base Time: 2017-10-01T12:00:00Z
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_92.9?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# -  
	# # 2018
	# # Use GLBv0.08/expt_93.0 for 1/1/2018 to 12/31/2018 or 2/18/2020, Base Time: 2018-01-01T12:00:00Z
	# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_93.0?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# - - -
	# # 2019-present (up to one year at a time)

	# # Use GLBy0.08 expt 93.0 for 04-Dec-2018 to present
	# # Native hycom .[ab] data converted to NetCDF at NRL, GLBy0.08 grid is 0.08 deg lon x 0.04 deg lat that covers 80 S to 90 N.
	# # Note from Amitava Guha: For latest years, you can change the $url to another experiment number as shown below:
	$url = "https://ncss.hycom.org/thredds/ncss/GLBy0.08/expt_93.0?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"

	# END OF USER SELECTION OF $url
	# - - -
	# - - -

	# - - -


	# Output file name
	$fileName = $time.ToString('yyyyMMdd_HH') + ".nc"
	$output = $resultDirectory + $fileName
	Try{

		# [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12

		# Creating a web client which has the download file functionality
		#WebProxy = New-Object System.Net.WebProxy("hoeprx01.na.xom.com:8080",$true)

		$WebClient = New-Object System.Net.WebClient
		#$WebClient.Proxy=$WebProxy
		$WebClient.DownloadFile($url,$output)

		Write-Host "Successfully downloaded:"  $fileName
		
		$error_flag = 0
	}
	Catch {
		Write-Host $_.Exception.Message`n
		$error_flag = 1
		Write-Host  "Retrying download of file:" $fileName " ...."
	}
}
}

Answer 2

[Zhiguo Mei]

Maybe you can try this "water_u = ncread(url, 'water_u',[1,1,1,1],[200,200,40,1]);"

Reduce the size of the variable you want read at once,and it always works for me

Answer 3

[Gabriel Ruiz-Martinez]

HI Amitava, thanks for sharing your Powershell script. Recently, I faced off the same problem as you. Using the NCTOOLBOX (Schilining et al., 2009), I did a Matlab script to download the HYCOM data from opeNDAP.

The code can find in Matlab File Exchange:

Exporting HYCOM currents data to txt and mat files

[Muhammad Nadzrin Nazri]

Dear Gabriel,

Can you guide me to download using MatLab as well? Instead of point interest I would like to download in form of area (south china sea):

North: 14

South: 0

East: 99

West: 117

I would like to download 1 hourly, surface elevation, U & V elevation data from jan 2019 - Aug 2020:

//ncss.hycom.org/thredds/ncss/grid/GLBy0.08/expt_93.0/sur

Thanks in advance.

[Gabriel Ruiz-Martinez]

Hi Muhammad,

I recognize you have to download points from an entire region. In this case, you should run my function for every point (or node). You should change the dataUrl and dataUrl1 variables with the Hindcast HYCOM dataset URL that you need to use.

My advice is to write a script where you save in vectors the inputs that function requires and then... call the function for every point.

Reading the OpeNDAP Quickstart doc, if I understood well, you can download a maximum of 50 MB of data. I think if you want to download an entire region of data, the data will exceed 50 MB, sending an error message. For this reason, maybe you need to download node by node. Taking into account this fact, I wrote the function.

Notice the URL you shared is NetcdfSubset URL, the function HYCOMnc2txtS requires the OPENDAP URL:

https://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0/uv3z

(I would think this the URL you need, but I don't sure)

Regards.