shell/math: return string error indicator, not integer

shell/math: return string error indicator, not integer

function                                             old     new   delta
expand_and_evaluate_arith                             87     106     +19
expand_one_var                                      1563    1570      +7
arith                                                 12      18      +6
evaluate_string                                      678     680      +2
arith_apply                                         1269    1271      +2
builtin_umask                                        133     132      -1
ash_arith                                            118      75     -43
expand_vars_to_list                                 1094    1038     -56
------------------------------------------------------------------------------
(add/remove: 0/0 grow/shrink: 5/3 up/down: 36/-100)           Total: -64 bytes

Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>