高空要素统计检验
<p>[TOC]</p>
<pre><code class="language-python">%matplotlib inline
%load_ext autoreload
%autoreload 2
import pandas as pd
import meteva.method as mem
import meteva.base as meb
import meteva.product as mpd
import meteva.perspact as mps # 透视分析模块
import datetime
import meteva
import numpy as np
import os
import copy</code></pre>
<pre><code>The autoreload extension is already loaded. To reload it, use:
%reload_ext autoreload</code></pre>
<h1>高空格点场统计检验</h1>
<p>在对数值模式和AI大模型数据进行检验时，经常要对长序列的多层次高空数据进行统计检验。这种长序列的统计非常耗时，并且在批量数据统计时，时常会因为各种原因导致程序中断，完成统计并非易事。改进的策略时及时的将中间结果进行保存，出错后再重复启动的时候不必从头再来。本模块的目标就是实现这样的功能。
该模块可以根据设定的实况和预报数据路径等参数，读取多个层次、多个时效、一段时间、多个模式的高空要素，生成数值型检验指标计算所需的中间量。如果程序因异常中断，重新启动后，程序能够重新加载此前已经统计过的大部分结果，对其余部分进行补齐。 之所以只能加载此前的大部分结果，是因为考虑到输出频次过高会影响效率，程序默认是完成约500个单时刻单时次单层次的中间量统计时 输出一次。</p>
<table>
<thead>
<tr>
<th style="text-align: left;">参数</th>
<th style="text-align: left;">说明</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;"><strong><font face="黑体" color=blue size = 5>para</font></strong></td>
<td style="text-align: left;">统计的参数</td>
</tr>
<tr>
<td style="text-align: left;"><font face="黑体" color=blue size=5>return</font></td>
<td style="text-align: left;">统计量中间结果输出到文件中，函数不返回结果</td>
</tr>
</tbody>
</table>
<p><strong> 参数示例和说明</strong> </p>
<pre><code class="language-python">para = {
&quot;mid_method&quot;:mem.tase, # 统计检验的总量计算函数，
&quot;grade_list&quot;:None, #二分类检验所需的等级参数
&quot;compare&quot;:None, #二分类检验所需的对比方式参数
&quot;middle_result_path&quot;: r&quot;H:\test_data\output\mps\tase_z.h5&quot;, #检验中间量数据的存储路径
&quot;begin_time&quot;: datetime.datetime(2018,8,1,0), #统计起始时刻， 它是预报的起报时间，而不是实况实况
&quot;end_time&quot;: datetime.datetime(2018,8,2,0), #统计结束时刻，它是预报的起报时间，而不是实况实况
&quot;time_type&quot;:&quot;UT&quot;, # 最终检验结果呈现时，采用北京时还是世界时，UT代表世界时，BT代表北京时
&quot;time_step&quot;:12, #起报时间间隔
&quot;grid&quot; : meteva.base.grid([70,140,0.25],[10,60,0.25]), # 检验区域
&quot;level_list&quot;: [ 850, 700, 500], #检验的层次
&quot;step&quot;:5, #masker间隔（单位：°）
&quot;recover&quot;:False, #False表示保留已经收集好的中间量，补齐未收集的数据，True表示删除已有的统计结果文件，重新开始收集
&quot;ob_data&quot;: { #支持多个实况数据
&quot;CRA40&quot;:{ #实况数据名称之一
&quot;dirm&quot;: r&quot;\\10.28.16.234\data2\AI\CRA40\2018\Z\LLL\YYYYMMDDHH.nc&quot;, # 实况场数据路径
&quot;read_method&quot;: meteva.base.read_griddata_from_nc, #读取数据的函数
&quot;read_para&quot;: {}, #读取数据的函数参数
&quot;time_type&quot;: &quot;UT&quot;, # 数据文件中的时间类型，UT代表世界时
&quot;multiple&quot;: 1, # 用于单位转换的系数，例如将比湿由kg/kg 转换成g/kg时， 该参数设置为10000
}
},
&quot;fo_data&quot;: {
&quot;ECMWF&quot;:{ #模式名称之一
&quot;dirm&quot;: r&quot;\\10.28.16.234\data2\AI\ECMWF\grib\YYMMDD\YYMMDDHH.TTT.grib&quot;, # 数据路径
&quot;dtime&quot;: [12, 240, 12], #检验的时效
&quot;read_method&quot;: meteva.base.read_griddata_from_grib, #读取数据的函数
&quot;read_para&quot;: {&quot;level_type&quot;:&quot;isobaricInhPa&quot;,&quot;value_name&quot;:&quot;gh&quot;}, #读取数据的函数参数
&quot;time_type&quot;: &quot;UT&quot;, # 预报数据时间类型是北京时，即08时起报
&quot;multiple&quot;: 1, # 用于单位转换的系数，例如将比湿由kg/kg 转换成g/kg时， 该参数设置为10000
&quot;move_fo_time&quot;:12, #是否对预报的时效进行平移，12 表示将1月1日08时的36小时预报转换成1月1日20时的24小时预报后参与对比
&quot;veri_by&quot;:&quot;self&quot; #self表示用自己的零场作为真值进行检验
},
&quot;NMC1&quot;: { #模式名称之
&quot;dirm&quot;: r&quot;\\10.28.16.177\NMC_Tsinghua_reforecast\cra40\YYYYMMDDHH.nc&quot;, #数据路径
&quot;dtime&quot;: [12, 240, 12], #检验时效
&quot;read_method&quot;: meteva.base.read_griddata_from_nc, #读取数据的函数
&quot;read_para&quot;: {&quot;value_name&quot;: &quot;GPH&quot;}, #读取数据的函数参数
&quot;time_type&quot;: &quot;UT&quot;,#预报数据时间类型是北京时，即08时起报
&quot;multiple&quot;:1, #用于单位转换的系数，例如文件数据的单位是位势米，通过乘以9.8可转换成位势
&quot;move_fo_time&quot;: 0,
&quot;veri_by&quot;: &quot;CRA40&quot; #指定某一个实况数据作为真值进行检验
},
},
}
</code></pre>
<h2>关于para的补充说明：</h2>
<ol>
<li>mid_method 参数常用的选项包括 mem.tase,mem.tmmsss,mem.hfmc等。如果要统计平均误差、平均绝对误差、均方根误差就选mem.tase；如果要计算二分类的指标，如TS、BIAS等就选择mem.hfmc； 如果要选相关系数就选mem.tmmsss，大部分情况下高空要素一般要计算距平相关系数，而非普通的相关系数。在meteva中计算距平相关系数有专门的ACC模块。</li>
<li>grade_list 和compare参数只有当mid_method = mem.hfmc时才需要设置，它们是二分类检验要用到的参数</li>
<li>统计形成的中间量数据文件中包含但未报仅包含begin_time 至end_time 时段内的数据，如果recover=False，它还会包含上一次运行留下来的统计数据。</li>
<li>time_step参数一般用于起报时间的间隔，例如全球模式一般是12小时间隔报一次。在调试阶段，time_step也可以设置得大一些，例如1200， 这样程序会跳跃着读取部分日期的数据进行统计，用户可以根据部分数据的统计结果来计算评分和绘图，如果检验结果合理，在重新将dtime 设置为12进行统计。避免花费很长时间完成统计后，结果是错误的，则又要重新返工，浪费大量时间。 对于非常长的时间序列，time_step 也未必一定要根据实际的预报起报时间间隔来进行设置，例如时间间隔是12小时，那将time_step 设置成60小时，也可以得到很有代表性的统计结果。此处只所以设置为60而不是48或72，是考虑到后面两种设置方法不能遍历到00和12时两种起报时间。</li>
<li>read_method对应的读取数据函数应该包含参数time,level,dtime，本模块的程序会自动循环采用不同的time,dtime,level参数调用read_method。read_para 中则不必包含time,dtime,level相关的内容，因为本程序会自动设置这些参数。</li>
<li>veri_by 用于指定检验实况，每个模式可用选择相同的实况，也可以选择不同的实况。</li>
</ol>
<h2>调用方法</h2>
<pre><code class="language-python"># 统计rmse，me,mae 相关的中间量
mps.middle_of_score(para)</code></pre>
<pre><code>\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
中间量统计程序运行完毕
中间结果已输出至H:\test_data\output\mps\tase_z.h5</code></pre>
<pre><code class="language-python">#加载中结果数据
df_mid = pd.read_hdf(para[&quot;middle_result_path&quot;])
print(df_mid)</code></pre>
<pre><code> level time dtime member id T E \
0 850 2018-08-01 12 ECMWF 30080 1446.777004 -8817.015932
0 850 2018-08-01 12 ECMWF 20100 1542.615732 -7113.460043
0 850 2018-08-01 12 ECMWF 40070 32.674467 -123.065719
0 850 2018-08-01 12 ECMWF 10120 39.392310 -62.056353
0 850 2018-08-01 12 ECMWF 30090 1446.777004 -2326.123738
.. ... ... ... ... ... ... ...
0 500 2018-08-02 240 NMC1 20080 1542.615732 19958.427028
0 500 2018-08-02 240 NMC1 10100 39.392310 404.537590
0 500 2018-08-02 240 NMC1 30070 36.169425 352.785362
0 500 2018-08-02 240 NMC1 20090 1542.615732 33132.055382
0 500 2018-08-02 240 NMC1 10110 39.392310 296.846196
A S
0 8848.364657 62897.818132
0 7347.701158 44004.340204
0 125.558343 655.077934
0 77.316545 248.773984
0 3373.974168 15554.391699
.. ... ...
0 19958.427028 315540.812378
0 404.537590 4536.645029
0 356.379313 6054.641830
0 33132.055382 737601.176270
0 296.846196 2334.247619
[16848 rows x 9 columns]</code></pre>
<pre><code class="language-python">#根据中间量结果绘制误差曲线
result = mps.score_df(df_mid,mem.rmse,g = [&quot;level&quot;,&quot;member&quot;,&quot;dtime&quot;],plot =&quot;line&quot;,ncol = 3,sup_title = &quot;RMSE&quot;)</code></pre>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=7dc1c8ab5294d0c89a428c863ced33f6&amp;file=file.png" alt="" /></p>
<p>上面是对比不同模式的rmse的实现方式。如果需要对比模式对副高的预报性能，则可以用二分类检验指标来实现</p>
<pre><code class="language-python"># 设置统计参数
para_hfmc = copy.deepcopy(para)
para_hfmc[&quot;mid_method&quot;] = mem.hfmc
para_hfmc[&quot;grade_list&quot;] = [5880,5920] # 副高区域
para_hfmc[&quot;middle_result_path&quot;] = r&quot;H:\test_data\output\mps\hfmc_500.h5&quot;
para_hfmc[&quot;level_list&quot;] = [500]
# 统计rmse，me,mae 相关的中间量
mps.middle_of_score(para_hfmc)</code></pre>
<pre><code>success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.024.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180802\18080200.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.036.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180802\18080212.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.048.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080300.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.060.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080312.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.072.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080400.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.084.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080412.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.096.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080500.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.108.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080512.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.120.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080600.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.132.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080612.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.144.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080700.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.156.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080712.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.168.grib
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180808\18080812.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.192.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080900.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.204.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080912.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.216.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081000.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.228.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081012.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.240.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180811\18081100.000.grib
Traceback (most recent call last):
File &quot;h:\task\develop\python\git\meteva\meteva\base\io\read_griddata.py&quot;, line 346, in read_griddata_from_grib
ds0 = xr.open_dataset(filename, engine=&quot;cfgrib&quot;, backend_kwargs={'filter_by_keys': filter_by_keys,&quot;indexpath&quot;: &quot;&quot;},)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\api.py&quot;, line 495, in open_dataset
backend_ds = backend.open_dataset(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 127, in open_dataset
store = CfGribDataStore(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 66, in __init__
self.ds = cfgrib.open_file(filename, **backend_kwargs)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 782, in open_file
index = open_fileindex(stream, indexpath, index_keys, filter_by_keys=filter_by_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 761, in open_fileindex
index = messages.FileIndex.from_indexpath_or_filestream(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 529, in from_indexpath_or_filestream
return cls.from_fieldset(filestream, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 378, in from_fieldset
return cls.from_fieldset_and_iteritems(fieldset, iteritems, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 391, in from_fieldset_and_iteritems
for field_id, raw_field in iteritems:
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 291, in __iter__
for message in self.itervalues():
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 267, in itervalues
with open(self.filestream.path, &quot;rb&quot;) as file:
FileNotFoundError: [Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180731\\18073112.252.grib'
[Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180731\\18073112.252.grib'
faild to read \\10.28.16.234\data2\AI\ECMWF\grib\180731\18073112.252.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180802\18080200.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.024.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180802\18080212.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.036.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080300.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.048.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080312.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.060.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080400.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.072.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080412.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.084.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080500.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.096.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080512.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.108.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080600.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.120.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080612.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.132.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080700.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.144.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080712.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.156.grib
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180808\18080812.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.180.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080900.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.192.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080912.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.204.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081000.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.216.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081012.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.228.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180811\18081100.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.240.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180811\18081112.000.grib
Traceback (most recent call last):
File &quot;h:\task\develop\python\git\meteva\meteva\base\io\read_griddata.py&quot;, line 346, in read_griddata_from_grib
ds0 = xr.open_dataset(filename, engine=&quot;cfgrib&quot;, backend_kwargs={'filter_by_keys': filter_by_keys,&quot;indexpath&quot;: &quot;&quot;},)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\api.py&quot;, line 495, in open_dataset
backend_ds = backend.open_dataset(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 127, in open_dataset
store = CfGribDataStore(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 66, in __init__
self.ds = cfgrib.open_file(filename, **backend_kwargs)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 782, in open_file
index = open_fileindex(stream, indexpath, index_keys, filter_by_keys=filter_by_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 761, in open_fileindex
index = messages.FileIndex.from_indexpath_or_filestream(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 529, in from_indexpath_or_filestream
return cls.from_fieldset(filestream, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 378, in from_fieldset
return cls.from_fieldset_and_iteritems(fieldset, iteritems, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 391, in from_fieldset_and_iteritems
for field_id, raw_field in iteritems:
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 291, in __iter__
for message in self.itervalues():
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 267, in itervalues
with open(self.filestream.path, &quot;rb&quot;) as file:
FileNotFoundError: [Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180801\\18080100.252.grib'
[Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180801\\18080100.252.grib'
faild to read \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080100.252.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180802\18080212.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.024.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080300.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.036.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180803\18080312.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.048.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080400.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.060.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180804\18080412.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.072.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080500.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.084.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180805\18080512.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.096.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080600.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.108.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180806\18080612.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.120.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080700.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.132.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180807\18080712.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.144.grib
\\10.28.16.234\data2\AI\ECMWF\grib\180808\18080800.000.grib not exist
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180808\18080812.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.168.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080900.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.180.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180809\18080912.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.192.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081000.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.204.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180810\18081012.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.216.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180811\18081100.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.228.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180811\18081112.000.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.240.grib
success read from \\10.28.16.234\data2\AI\ECMWF\grib\180812\18081200.000.grib
Traceback (most recent call last):
File &quot;h:\task\develop\python\git\meteva\meteva\base\io\read_griddata.py&quot;, line 346, in read_griddata_from_grib
ds0 = xr.open_dataset(filename, engine=&quot;cfgrib&quot;, backend_kwargs={'filter_by_keys': filter_by_keys,&quot;indexpath&quot;: &quot;&quot;},)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\api.py&quot;, line 495, in open_dataset
backend_ds = backend.open_dataset(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 127, in open_dataset
store = CfGribDataStore(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\xarray\backends\cfgrib_.py&quot;, line 66, in __init__
self.ds = cfgrib.open_file(filename, **backend_kwargs)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 782, in open_file
index = open_fileindex(stream, indexpath, index_keys, filter_by_keys=filter_by_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\dataset.py&quot;, line 761, in open_fileindex
index = messages.FileIndex.from_indexpath_or_filestream(
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 529, in from_indexpath_or_filestream
return cls.from_fieldset(filestream, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 378, in from_fieldset
return cls.from_fieldset_and_iteritems(fieldset, iteritems, index_keys, computed_keys)
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 391, in from_fieldset_and_iteritems
for field_id, raw_field in iteritems:
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 291, in __iter__
for message in self.itervalues():
File &quot;C:\Users\admin\.conda\envs\python38\lib\site-packages\cfgrib\messages.py&quot;, line 267, in itervalues
with open(self.filestream.path, &quot;rb&quot;) as file:
FileNotFoundError: [Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180801\\18080112.252.grib'
[Errno 2] No such file or directory: '\\\\10.28.16.234\\data2\\AI\\ECMWF\\grib\\180801\\18080112.252.grib'
faild to read \\10.28.16.234\data2\AI\ECMWF\grib\180801\18080112.252.grib
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080112.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080200.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080212.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080300.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080312.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080400.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080412.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080500.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080512.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080600.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080612.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080700.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080712.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080800.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080812.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080900.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080912.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081000.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081012.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081100.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080100.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080200.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080212.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080300.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080312.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080400.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080412.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080500.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080512.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080600.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080612.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080700.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080712.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080800.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080812.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080900.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080912.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081000.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081012.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081100.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081112.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080112.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080212.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080300.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080312.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080400.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080412.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080500.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080512.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080600.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080612.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080700.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080712.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080800.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080812.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080900.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018080912.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081000.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081012.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081100.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081112.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
success read from \\10.28.16.234\data2\AI\CRA40\2018\Z\500\2018081200.nc
success read from \\10.28.16.177\NMC_Tsinghua_reforecast\cra40\2018080200.nc
中间量统计程序运行完毕
中间结果已输出至H:\test_data\output\mps\hfmc_500.h5</code></pre>
<pre><code class="language-python">#加载中结果数据
df_mid = pd.read_hdf(para_hfmc[&quot;middle_result_path&quot;])
print(df_mid)</code></pre>
<pre><code> level time dtime member id grade H F M C
0 500 2018-08-01 12 ECMWF 30080 5880 0.0 0.0 0.0 1600.0
1 500 2018-08-01 12 ECMWF 30080 5920 0.0 0.0 0.0 1600.0
0 500 2018-08-01 12 ECMWF 20100 5880 0.0 0.0 0.0 1600.0
1 500 2018-08-01 12 ECMWF 20100 5920 0.0 0.0 0.0 1600.0
0 500 2018-08-01 12 ECMWF 40070 5880 0.0 0.0 0.0 40.0
.. ... ... ... ... ... ... ... ... ... ...
1 500 2018-08-02 240 NMC1 30070 5920 0.0 0.0 0.0 40.0
0 500 2018-08-02 240 NMC1 20090 5880 0.0 0.0 0.0 1600.0
1 500 2018-08-02 240 NMC1 20090 5920 0.0 0.0 0.0 1600.0
0 500 2018-08-02 240 NMC1 10110 5880 0.0 0.0 0.0 40.0
1 500 2018-08-02 240 NMC1 10110 5920 0.0 0.0 0.0 40.0
[10944 rows x 10 columns]</code></pre>
<pre><code class="language-python">#根据实况和预报的频次（累计格点数）的对比
result = mps.score_df(df_mid,mem.ob_fo_hc,g = [&quot;grade&quot;,&quot;member&quot;,&quot;dtime&quot;],plot =&quot;line&quot;,ncol = 2,sup_title = &quot;RMSE&quot;)</code></pre>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=e835a194aff3b5f3667b906c0e7f57ba&amp;file=file.png" alt="" /></p>
<p>如果数据时段很长，实况的副高累计格点数随时效基本是不变化的，但上面的数据集只包含一小段时间的数据，因此实况的副高累计格点数随时效变化非常剧烈。从上面的结果还可以看出，两种模式对副高面积（格点数）的预报都是偏低的。</p>