hed-tutorial-for-document-scanningnode
(py27tf) bash-3.2$ ls
README.md
checkpoint
const.py
const.pyc
dataset
evaluate_hed.py
freeze_model.py
generate_training_dataset
hed_net.py
hed_net.pyc
how_to_build_tensorflow_and_change_namespace_of_protobuf.txt
input_pipeline.py
input_pipeline.pyc
ios_demo
mobilenet.py
mobilenet.pyc
preprocess_generate_training_dataset.py
run_preprocess.sh
run_train.sh
sample_images
test_image
train_hed.py
util.py
util.pyc
(py27tf) bash-3.2$ pwd
/Users/taily/githubproj/hed-tutorial-for-document-scanning
(py27tf) bash-3.2$ gshuf ./dataset/generate_sample_by_ios_image_size_256_256_thickness_0.2.csv > ./dataset/temp.txt
(py27tf) bash-3.2$ gshuf ./dataset/temp.txt > ./dataset/generate_sample_by_ios_image_size_256_256_thickness_0.2.csv
(py27tf) bash-3.2$ sh run_train.sh
TensorFlow Version: 1.9.0
###########################################
###########################################
dataset_root_dir is: dataset
os.path.join(FLAGS.dataset_root_dir, '') is: dataset/
csv_path is: dataset/generate_sample_by_ios_image_size_256_256_thickness_0.2.csv
checkpoint_dir is: ./checkpoint
debug_image_dir is: ./debug_output_image
###########################################
###########################################
image_tensor shape is: (4, 256, 256, 3)
dsn_fuse shape is: (4, 256, 256, 1)
cost shape is: ()
############################################################
2019-01-22 21:46:42.382968: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-01-22 21:46:42.383826: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 8. Tune using inter_op_parallelism_threads for best performance.
restore from latest checkpoint file : ./checkpoint/hed.ckpt-130
Start train...
batch_size is: 4
iterations is: 90000000
display-step is: 5
learning-rate is: 0.0005
++ use batch norm
2019-01-22 21:46:50.439666: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: dataset/Documents/j0AHxUSqtL_random_size_118_101_1_annotation_thresh_gray.png; No such file or directory
2019-01-22 21:46:50.439669: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: dataset/Documents/j0AHxUSqtL_random_size_118_101_1_color.jpg; No such file or directory
Traceback (most recent call last):
File "train_hed.py", line 205, in <module>
feed_dict=feed_dict_to_use)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
Caused by op u'shuffle_batch', defined at:
File "train_hed.py", line 98, in <module>
FLAGS.batch_size)
File "/Users/taily/githubproj/hed-tutorial-for-document-scanning/input_pipeline.py", line 71, in fix_size_image_pipeline
min_after_dequeue = min_after_dequeue)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1300, in shuffle_batch
name=name)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 846, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 483, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3480, in queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/anaconda2/envs/py27tf/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
(py27tf) bash-3.2$
(py27tf) bash-3.2$ gshuf ./dataset/Documents.csv > ./dataset/temp.txt (py27tf) bash-3.2$ gshuf ./dataset/temp.txt > ./dataset/Documents.csv (py27tf) bash-3.2$ sh run_train.sh TensorFlow Version: 1.9.0 ########################################### ########################################### dataset_root_dir is: dataset os.path.join(FLAGS.dataset_root_dir, '') is: dataset/ csv_path is: dataset/Documents.csv checkpoint_dir is: ./checkpoint debug_image_dir is: ./debug_output_image ########################################### ########################################### image_tensor shape is: (4, 256, 256, 3) dsn_fuse shape is: (4, 256, 256, 1) cost shape is: () ############################################################ 2019-01-22 21:58:17.180934: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2019-01-22 21:58:17.181668: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 8. Tune using inter_op_parallelism_threads for best performance. restore from latest checkpoint file : ./checkpoint/hed.ckpt-130 Start train... batch_size is: 4 iterations is: 90000000 display-step is: 5 learning-rate is: 0.0005 ++ use batch norm Step: 0, Current Mean Loss: 0.00144546641968 Step: 5, Current Mean Loss: 0.00107343564741 Step: 10, Current Mean Loss: 0.00129083264619 Step: 15, Current Mean Loss: 0.00104159826878 Step: 20, Current Mean Loss: 0.00137554691173 Step: 25, Current Mean Loss: 0.00208519585431 Step: 30, Current Mean Loss: 0.00133184040897 Step: 35, Current Mean Loss: 0.00169059564359 Step: 40, Current Mean Loss: 0.00172041752376 Step: 45, Current Mean Loss: 0.0021527602803 Step: 50, Current Mean Loss: 0.00364822335541 Step: 55, Current Mean Loss: 0.00226333946921 Step: 60, Current Mean Loss: 0.00129697239026 Step: 65, Current Mean Loss: 0.0053047221154 Step: 70, Current Mean Loss: 0.00146268832032 Step: 75, Current Mean Loss: 0.00175709614996 Step: 80, Current Mean Loss: 0.00176999950781 Step: 85, Current Mean Loss: 0.00185493286699 Step: 90, Current Mean Loss: 0.0024897470139 Step: 95, Current Mean Loss: 0.00242765760049 Step: 100, Current Mean Loss: 0.00135294231586 Step: 105, Current Mean Loss: 0.0012677735649 Step: 110, Current Mean Loss: 0.00466330628842 Step: 115, Current Mean Loss: 0.00320365303196 Step: 120, Current Mean Loss: 0.00117148994468 Step: 125, Current Mean Loss: 0.00102473050356 Step: 130, Current Mean Loss: 0.00296807009727 Step: 135, Current Mean Loss: 0.00162225530948 Step: 140, Current Mean Loss: 0.00163191650063 Step: 145, Current Mean Loss: 0.00130305998027 Step: 150, Current Mean Loss: 0.00278477417305 Step: 155, Current Mean Loss: 0.00198156945407 Step: 160, Current Mean Loss: 0.00142371770926 Step: 165, Current Mean Loss: 0.00126392161474 Step: 170, Current Mean Loss: 0.00267102662474