Bug 30975

Summary: (spec %check) automatically detect whether a test has failed
Product: Sisyphus Reporter: Ivan Zakharyaschev <imz>
Component: drakonAssignee: Ivan Zakharyaschev <imz>
Status: NEW --- QA Contact: qa-sisyphus
Severity: minor    
Priority: P3 CC: lav
Version: unstable   
Hardware: all   
OS: Linux   

Description Ivan Zakharyaschev 2015-04-27 17:38:47 MSK 
drakon-1.22-alt1

I can't detect automatically whether there has been a failure in the
unittests called from %check in the RPM spec.

In older versions of drakon, I grepped the output and checked whether
there are only the names of the tests there --
http://git.altlinux.org/people/imz/packages/?p=drakon.git;a=blob;f=drakon.spec;h=b26edeaacd9a60b0ebbd81145b249f023b56fc33;hb=848248d8591ae11be5e9ca29ad098f55170c4dd7#l70
.

That was needed because the exit code was 0 even if it failed (because of a not found library).

In newer versions (1.22 -- the last one which doesn't require tcl8.6),
there are tests which output something (see below), so I don't know
how to check whether all tests were successful.

We should come up with a way to check this or to modify the source code so
that it exits with different exit codes...

As for now, I have disabled %check in the RPM spec.

Executing(%check): /bin/sh -e /usr/src/tmp/rpm-tmp.13657
+ umask 022
+ /bin/mkdir -p /usr/src/RPM/BUILD
+ cd /usr/src/RPM/BUILD
+ cd drakon-1.22
+ pushd unittest/
~/RPM/BUILD/drakon-1.22/unittest ~/RPM/BUILD/drakon-1.22
+ ./unittest.tcl
+ tee check.log
flag_on_test
repeat_test
zip_test
wrap_test
invoke_test
invoke_all_test
vec2.addxy.test
swap_test
rectangles_intersect_test
rectangles_on_axis_test
touching_side_test
push_rect_test
add_border_test
hit_rectangle_test
move_rectangle_test
make_rect_test
snap_test
intervals_touch_test
intervals_intersect_test
map_test
filter_test
lfilter_test
lfilter_user_test
snap_delta_test
sql_escape_test
char_set_test
unpack_test
line_hit_box_test
box_cut_line_vertical_test
box_cut_line_horizontal_test
intersect_lines_updown_test
intersect_lines_leftright_test
contains_test
remove_test
init_cap_test
replace_extension_test
generate_structure_test
put_value_test
have_intersection_test
is_variable_test
add_range_test
append_not_empty_test
subtract_test
dict_get_safe_test
art_test
diagram: 2 rows
id 1 name jiafei description {vud boss}
id 2 name alan description {}
diagram: 2 rows
id 1 name jiafei description {vud boss}
id 2 name alan description {}
open_model_ok
create_model_ok
open_model_fail
create_model_fail
save_as_model
sql_build
apply_test
next_key_test
zoomup_test
zoomdown_test
zoom_vertices_test
diagram_info_test
undo_test
switch_to_item_test
insert_action_test
take_selection_from_shadow_test
take_shapes_from_shadow_test
change_text_test
layer_unlayer
zplace_test
add_to_canvas_test
create_prim_test
insert_delete
find_items_test
hit_test
hit_handle_test
selection_test
action_handle_test
resize_test
shadow_selection_test
to_tokens_test
find_occurences_test
find_token_occurences_test
replace_occurences_test
replace_one_test
replace_test
find_all_test
extract_auto_test
extract_auto_test2
branch_trouble
auto_test
extract_sections_test
extract_return_type_test
extract_tcl_signature_test
extract_for_test
extract_foreach_test
extract_cpp_signature_test
extract_java_signature_test
extract_java_class_test
push_test
pop_test
nsorter_test
extract_copying_test
extract_class_name_test
get_alt_item_test
nogoto_test
noggen_test
extract_tables_test
diff_array_test
parse_message_signature_test
algo2_test
after_others_test
find_common_point_test
merge.simple_test
line.arrow_test
arrow.line_test
merge.segments_test
merge.on.1.line_test
icons.touch_test
cut.simple_test
segment.cut.segment_test
lines.cut_test
get.only_test
connection_build_test
class_build_error_test
class_build_success_test
automaton_test
sm_no_branches.drn
Diagram 'state machine' is a state machine and must be a silhouette.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id "" $message"
    (procedure "bad_diagram" line 3)
    invoked from within
"bad_diagram $diagram_id  "Diagram '$name' is a state machine and must
be a silhouette.""
    (procedure "has_branches" line 15)
    invoked from within
"has_branches $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: Diagram 'state machine' is a state machine and must be
a silhouette.
sm_no_receive.drn
No 'receive' statements found.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id "" $message"
    (procedure "bad_diagram" line 3)
    invoked from within
"bad_diagram $diagram_id  "No 'receive' statements found.""
    (procedure "build_machine" line 9)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: No 'receive' statements found.
sm_receive_middle.drn
A 'receive' icon must be first in the branch.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $receive  "A 'receive' icon must be first in the branch.""
    (procedure "receives_are_first" line 21)
    invoked from within
"receives_are_first $gdb $receives"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: item 98: A 'receive' icon must be first in the branch.
sm_receive_end.drn
Exit branch should not start with a 'receive'.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $header  "Exit branch should not start with a 'receive'.""
    (procedure "all_start_with_receive" line 20)
    invoked from within
"all_start_with_receive $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
State machine: item 10: Exit branch should not start with a 'receive'.
sm_no_receive_middle.drn
State branch should start with a 'receive'.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $header  "State branch should start with a 'receive'.""
    (procedure "all_start_with_receive" line 36)
    invoked from within
"all_start_with_receive $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: item 19: State branch should start with a 'receive'.
+ egrep -v '^[[:alnum:]_.]*$' check.log
diagram: 2 rows
id 1 name jiafei description {vud boss}
id 2 name alan description {}
diagram: 2 rows
id 1 name jiafei description {vud boss}
id 2 name alan description {}
Diagram 'state machine' is a state machine and must be a silhouette.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id "" $message"
    (procedure "bad_diagram" line 3)
    invoked from within
"bad_diagram $diagram_id  "Diagram '$name' is a state machine and must
be a silhouette.""
    (procedure "has_branches" line 15)
    invoked from within
"has_branches $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: Diagram 'state machine' is a state machine and must be
a silhouette.
No 'receive' statements found.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id "" $message"
    (procedure "bad_diagram" line 3)
    invoked from within
"bad_diagram $diagram_id  "No 'receive' statements found.""
    (procedure "build_machine" line 9)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: No 'receive' statements found.
A 'receive' icon must be first in the branch.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $receive  "A 'receive' icon must be first in the branch.""
    (procedure "receives_are_first" line 21)
    invoked from within
"receives_are_first $gdb $receives"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: item 98: A 'receive' icon must be first in the branch.
Exit branch should not start with a 'receive'.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $header  "Exit branch should not start with a 'receive'.""
    (procedure "all_start_with_receive" line 20)
    invoked from within
"all_start_with_receive $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
State machine: item 10: Exit branch should not start with a 'receive'.
State branch should start with a 'receive'.
    while executing
"error $message"
    (procedure "gen::report_error" line 3)
    invoked from within
"gen::report_error $diagram_id $items $message"
    (procedure "bad_vertex" line 9)
    invoked from within
"bad_vertex $gdb $header  "State branch should start with a 'receive'.""
    (procedure "all_start_with_receive" line 36)
    invoked from within
"all_start_with_receive $gdb $diagram_id"
    (procedure "build_machine" line 12)
    invoked from within
"build_machine $gdb $diagram_id $callbacks "
    (procedure "sma::extract_machine" line 22)
    invoked from within
"sma::extract_machine $gdb $callbacks "
    (procedure "gen_cs::generate" line 7)
    invoked from within
"$generator $db $gdb $filename"
    (procedure "p.do_generate" line 8)
    invoked from within
"p.do_generate $generator $language $dst_filename "
state machine: item 19: State branch should start with a 'receive'.
+ echo 'It seems that a test printed an error message.'
It seems that a test printed an error message.
+ exit 1
error: Bad exit status from /usr/src/tmp/rpm-tmp.13657 (%check)
Comment 1 Ivan Zakharyaschev 2015-04-27 17:41:54 MSK 
I've found the answer.

The answer is in this change to unittest.tcl , which puts "success" at
the end -- https://github.com/stepan-mitkin/drakon_editor/commit/2473affd4eb5be5d03c390f36f557754ae472ecd#diff-c3e047f69439ee7aa529996a7e547dc5R76
.

We should switch the gear repo to refer to the new upstream Git repo, and cherry-pick that change.

(We still cannot use the latest release of drakon because it requires tcl 8.6. -- https://bugzilla.altlinux.org/show_bug.cgi?id=30407#c2 )