Datatype Limitation  
Author Message
Tech80inc





PostPosted: Mon Aug 08 12:45:35 CDT 2005 Top

Visual Basic >> Datatype Limitation

Hi,

I used Double var to get the following exp claculated,
7064.93105099284 * (1 - 2 / 100) and I got the result as,
6923.63242997298. But When I used Calculator to do the evaluation of the same
exp, I got
6923.6324299729832, where I got 2 more digits in the decimal part.

No my question is whether this is a limitation of VB or is there a
solution for the same.

Jiju Joseph

Visual Studio322  
 
 
Rick





PostPosted: Mon Aug 08 12:45:35 CDT 2005 Top

Visual Basic >> Datatype Limitation > I used Double var to get the following exp claculated,
> 7064.93105099284 * (1 - 2 / 100) and I got the result as,
> 6923.63242997298. But When I used Calculator to do the evaluation of
the same
> exp, I got
> 6923.6324299729832, where I got 2 more digits in the decimal part.
>
> No my question is whether this is a limitation of VB or is there a
> solution for the same.

It is not a limitation of VB; but, rather, a limitation of the data type
you chose to use to hold your number. A Double can only hold (display
actually), 15 significant digits, so that's what you got. IF you really
need ALL the decimal digits for your program (hard to imagine any
program where numbers in the 16th and 17th decimal position will
actually matter significantly in your calculations), then you will have
to use a Variant and convert at least one of your numbers to the Decimal
data type.

Debug.Print CDec(7064.93105099284) * (1 - 2 / 100)

The Decimal data type can work up to 28 or 29 significant digits;
however, there are limitation to this method if you plan on chaining the
answer to other calculations. There is no overflow protection and
built-in VB functions (such as Sin) will round your number down to the
Double data type it is designed to work with. Personally, I'd just stick
to the values the Double provides for you and leave it at that.

Rick


 
 
Ralph





PostPosted: Mon Aug 08 12:53:38 CDT 2005 Top

Visual Basic >> Datatype Limitation


> > I used Double var to get the following exp claculated,
> > 7064.93105099284 * (1 - 2 / 100) and I got the result as,
> > 6923.63242997298. But When I used Calculator to do the evaluation of
> the same
> > exp, I got
> > 6923.6324299729832, where I got 2 more digits in the decimal part.
> >
> > No my question is whether this is a limitation of VB or is there a
> > solution for the same.
>
> It is not a limitation of VB; but, rather, a limitation of the data type
> you chose to use to hold your number. A Double can only hold (display
> actually), 15 significant digits, so that's what you got. IF you really
> need ALL the decimal digits for your program (hard to imagine any
> program where numbers in the 16th and 17th decimal position will
> actually matter significantly in your calculations), then you will have
> to use a Variant and convert at least one of your numbers to the Decimal
> data type.
>
> Debug.Print CDec(7064.93105099284) * (1 - 2 / 100)
>
> The Decimal data type can work up to 28 or 29 significant digits;
> however, there are limitation to this method if you plan on chaining the
> answer to other calculations. There is no overflow protection and
> built-in VB functions (such as Sin) will round your number down to the
> Double data type it is designed to work with. Personally, I'd just stick
> to the values the Double provides for you and leave it at that.
>
> Rick
>

Rick,

I wonder if it useful to occasionally remind programmers that NASA went to
the moon and back with pi never figured beyond 3 places and no calculations
over 4. <g>

-ralph